Information processing method and device

Abstract

The embodiment of the invention discloses an information processing method and device, relates to the technical field of media, and is used for improving the quality of image feature points. The method comprises: acquiring sensor information, the sensor information comprising a target acceleration and a target angular velocity, the target acceleration being an acceleration measured by an inertial measurement unit, and the target angular velocity being an angular velocity measured by an IMU; determining mapping information according to the sensor information, wherein the mapping information is used for representing a mapping relation between a current frame image block and a historical frame image block; and determining feature points of a current frame image block according to the mapping information, wherein the current frame image block is any image block in the current frame image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of media technology, and in particular to information processing methods and devices. Background Art

[0002] Simultaneous localization and mapping (SLAM) refers to the process by which a moving object calculates its own position based on sensor information while simultaneously building a map of its surroundings. Currently, SLAM is primarily used in robotics, virtual reality, and augmented reality. SLAM's uses include sensor positioning, subsequent path planning, and scene understanding.

[0003] In the simultaneous localization and mapping (SLAM) technology, feature points of two adjacent frames are obtained and matched to establish the correspondence between the feature points in the two images, and then the camera pose is estimated. In order to obtain the affine transformation matrix between the two frames, it is necessary to ensure that there are a certain number and quality of successfully matched feature points between the two frames.

[0004] Therefore, how to extract high-quality image feature points is an important factor affecting SLAM accuracy. Summary of the Invention

[0005] The embodiments of the present application provide an information processing method and apparatus for improving the quality of image feature points. To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an information processing method, which includes: obtaining sensor information, the sensor information including target acceleration and target angular velocity, the target acceleration being the acceleration measured by an inertial measurement unit (IMU), and the target angular velocity being the angular velocity measured by the IMU; determining mapping information based on the sensor information, the mapping information being used to characterize the mapping relationship between a current frame image block and a historical frame image block; and determining feature points of the current frame image block based on the mapping information, the current frame image block being any image block in the current frame image.

[0007] The method provided in the embodiments of the present application can adaptively control the number of feature points based on mapping information, ensuring that the number of feature points is neither too many nor too few, controlling the total number of feature points and reducing the cost and computational complexity of feature point extraction. This ensures that the number (or distribution) of feature points in corresponding areas of the current frame and historical frames (e.g., the previous and next frames) are relatively consistent, ensuring a sufficient number and quality of successfully matched feature points, thereby improving the quality of image feature points.

[0008] In one possible implementation, motion information can be determined based on the above-mentioned sensor information, where the above-mentioned motion information includes a target translation amount and a target quaternion, where the above-mentioned target translation amount is the translation amount of the above-mentioned current frame image, and the above-mentioned target quaternion is the quaternion corresponding to the rotation matrix of the above-mentioned current frame image; the above-mentioned mapping information is determined based on the above-mentioned motion information.

[0009] The method provided in the embodiment of the present application can determine motion information based on the sensor, find the mapping relationship between the corresponding image blocks of the previous and next frames, and make the feature point distribution of the corresponding areas of the previous and next frames (matching means matching the feature points of the corresponding areas) consistent according to the mapping relationship between the image blocks of the previous and next frames, which can improve the matching effect and improve the robustness of the motion.

[0010] In a possible implementation, the target image block of the current frame image block can be determined based on the mapping information, where the target image block is the image block in the historical frame image that corresponds to the current frame image block. The feature points of the current frame image block can be determined based on the target information of the target image block, where the target information includes the number of feature points and a feature point extraction threshold.

[0011] The method provided in the embodiments of the present application can adaptively control the number of feature points based on the number of feature points in the image blocks of the historical frame image and the feature point extraction threshold, so that the number of feature points is not too large or too small, the total number of feature points is controlled, and the cost and computational complexity of feature point extraction are reduced. The number (distribution) of feature points in corresponding areas of the current frame and the historical frames (such as the previous and next frames) are relatively consistent, ensuring a sufficient number and quality of feature points for successful matching, thereby improving the quality of the image feature points.

[0012] In a possible implementation, a feature point extraction threshold of the current frame image block may be determined based on target information of the target image block; and feature points of the current frame image block may be determined based on the feature point extraction threshold of the current frame image block.

[0013] The method provided in the embodiments of the present application can adaptively control the feature point extraction threshold of the current frame image block based on the number of feature points in the image block of the historical frame image and the feature point extraction threshold, so as to adaptively control the number of feature points in the current frame image block, so that the number of feature points is not too many or too few, control the total number of feature points, and reduce the cost and computational complexity of feature point extraction. The number (distribution) of feature points in the corresponding areas of the current frame and the historical frame (such as the previous and next frames) is relatively consistent, which can ensure a sufficient number and quality of successfully matched feature points, thereby improving the quality of the image feature points.

[0014] In a possible implementation, when the number of feature points of the target image block is less than a first threshold, the feature point extraction threshold of the target image block may be reduced to obtain the feature point extraction threshold of the current frame image block.

[0015] It is understandable that if the number of feature points of the target image block is less than the first threshold, it means that the target image block has too few feature points. Therefore, the feature point extraction threshold needs to be reduced to increase the number of feature points of the current frame image block.

[0016] In a possible implementation, when the number of feature points of the target image block is greater than a second threshold, the feature point extraction threshold of the target image block may be increased to obtain the feature point extraction threshold of the current frame image block.

[0017] It is understandable that if the number of feature points of the target image block is greater than the second threshold, it means that the target image block has too few feature points. Therefore, the feature point extraction threshold needs to be increased to reduce the number of feature points of the current frame image block.

[0018] In a second aspect, an embodiment of the present application provides an information processing device, which includes: a transceiver unit and a processing unit; the transceiver unit is used to obtain sensor information, the sensor information includes target acceleration and target angular velocity, the target acceleration is the acceleration measured by the IMU, and the target angular velocity is the angular velocity measured by the IMU; the processing unit is used to determine mapping information based on the sensor information, the mapping information is used to characterize the mapping relationship between the current frame image block and the historical frame image block; the processing unit is used to determine the feature points of the current frame image block based on the mapping information, the current frame image block is any image block in the current frame image.

[0019] In one possible implementation, the processing unit is specifically used to: determine motion information based on the sensor information, the motion information including target translation and target quaternion, the target translation being the translation of the current frame image, and the target quaternion being the quaternion corresponding to the rotation matrix of the current frame image; and determine the mapping information based on the motion information.

[0020] In a possible implementation, the processing unit is specifically used to: determine the target image block of the current frame image block based on the mapping information, where the target image block is the image block in the historical frame image that corresponds to the current frame image block; and determine the feature points of the current frame image block based on the target information of the target image block, where the target information includes the number of feature points and a feature point extraction threshold.

[0021] In a possible implementation, the processing unit is specifically configured to: determine a feature point extraction threshold of the current frame image block according to target information of the target image block; and determine feature points of the current frame image block according to the feature point extraction threshold of the current frame image block.

[0022] In one possible implementation, the processing unit is specifically configured to: when the number of feature points of the target image block is less than a first threshold, reduce the feature point extraction threshold of the target image block to obtain the feature point extraction threshold of the current frame image block; and when the number of feature points of the target image block is greater than a second threshold, increase the feature point extraction threshold of the target image block to obtain the feature point extraction threshold of the current frame image block.

[0023] In a third aspect, an embodiment of the present application further provides an information processing device, which includes: at least one processor, which, when the at least one processor executes program code or instructions, implements the above method in the above first aspect or any possible implementation thereof.

[0024] Optionally, the information processing device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0025] In a fourth aspect, embodiments of the present application further provide a chip comprising: an input interface, an output interface, and at least one processor. Optionally, the chip further comprises a memory. The at least one processor is configured to execute code in the memory. When the at least one processor executes the code, the chip implements the method described in the first aspect or any possible implementation thereof.

[0026] Optionally, the chip may also be an integrated circuit.

[0027] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program includes methods for implementing the above-mentioned first aspect or any possible implementation thereof.

[0028] In a sixth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to implement the above-mentioned method in the above-mentioned first aspect or any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0031] Figure 2 A flowchart of an information processing method provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of a current frame image provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of the structure of an information processing device provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of the structure of another information processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the embodiments of this application.

[0038] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0039] The terms "first" and "second" and so on in the description and drawings of the embodiments of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0040] Furthermore, the terms "including," "having," and any variations thereof, mentioned in the description of the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0041] It should be noted that in the description of the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as having priority or advantage over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0042] Before introducing the technical solutions of this application, some of the terms involved in this application are explained. The following related explanations can be combined with the technical solutions of the embodiments of this application as optional solutions, and they all fall within the scope of protection of the embodiments of this application. The embodiments of this application include at least part of the following contents.

[0043] In SLAM, feature points of two adjacent frames are obtained, and the corresponding relationship between the feature points in the two images is established through feature point matching, and then the camera pose estimation is performed. In order to obtain the affine transformation matrix between the two frames, it is necessary to ensure that there are a certain number and quality of feature points that are successfully matched between the two frames.

[0044] To this end, an embodiment of the present application provides an information processing method for improving the quality of image feature points.

[0045] The technical solutions provided in the embodiments of the present application can be applied to electronic devices. The electronic devices can be mobile phones, robot vacuums, wearable devices such as smart watches, virtual reality (VR) glasses, sports bracelets, etc., as well as self-driving cars, drones, and other devices.

[0046] Figure 1 FIG. 1 shows a possible form of existence of the above electronic device. Figure 1 The electronic device 100 shown may include an inertial measurement unit 101 , an image acquisition unit 102 , and a processor 110 .

[0047] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0048] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0049] The processor 110 may further include a memory for storing instructions and data. In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0050] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present invention, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0051] It is understandable that Figure 1The structure of the electronic device 100 shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] Exemplarily, the electronic device 100 may further include: an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 151, a wireless communication module 152, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a SIM card interface 195. The sensor module 180 may include a gyroscope sensor 180A, an acceleration sensor 180B, a magnetic field sensor 180C, an electric field sensor 180D, a pressure sensor 180E, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, and a rotation axis sensor 180M.

[0053] Of course, the electronic device 100 may also include other sensors, such as a temperature sensor, a distance sensor, an ambient light sensor, an air pressure sensor, a bone conduction sensor, and the like.

[0054] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0055] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0056] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0057] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a touch sensor, a video codec, a GPU, a display screen 194, and an application processor.

[0058] The ISP is used to process data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0059] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. It should be understood that in the description of the embodiment of the present invention, the image in RGB format is used as an example, and the embodiment of the present invention does not limit the image format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0060] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0061] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0062] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area.

[0063] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0064] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall effect sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when photographing a scene, the electronic device 100 can use the distance sensor 180F to measure distance for rapid focusing. The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application lock, fingerprint photo taking, fingerprint call answering, etc. The temperature sensor 180J is used to detect temperature. The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The bone conduction sensor 180M can obtain vibration signals. The audio module 170 can parse the voice signal based on the vibration signal of the vocal bone block obtained by the bone conduction sensor 180M to realize the voice function.

[0065] The buttons 190 include a power button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The electronic device 100 can receive button input and generate key signal input related to the user settings and function control of the electronic device 100. The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0066] Figure 2 An information processing method provided by an embodiment of the present application is shown. Figure 2 As shown, the method includes:

[0067] S201: Acquire sensor information.

[0068] The sensor information includes target acceleration and target angular velocity, wherein the target acceleration is the acceleration measured by the IMU, and the target angular velocity is the angular velocity measured by the IMU.

[0069] For example, the target angular velocity Can satisfy:

[0070]

[0071] Among them, wt is the true value of angular velocity (i.e., the angular velocity of the IMU actual carrying body), bwt is the angular velocity bias noise, and nw is the angular velocity Gaussian white noise.

[0072] For example, if the actual IMU carrier is a sweeping robot, wt is the angular velocity of the sweeping robot.

[0073] For another example, if the actual carrier of the IMU is a smart car, wt is the angular velocity of the smart car.

[0074] For example, the target acceleration Can satisfy:

[0075]

[0076] in, is the rotation matrix from the IMU coordinate system to the world coordinate system at time t, at is the true value of acceleration (i.e., the acceleration of the actual carrying body), bat is the acceleration bias noise, na is the acceleration Gaussian white noise, and gw is the acceleration of the IMU in the world coordinate system w.

[0077] The above gw can be 9.8m / s2.

[0078] above It can be obtained through calibration. For example, the camera and IMU can be fixedly connected on the same plane to form a camera and IMU sensor. The camera and IMU sensor can be calibrated to obtain the sampling period of the camera and IMU, the camera intrinsic parameter K and the rotation matrix from the world coordinate system to the IMU coordinate system at time t. right Taking the inverse we get

[0079] S202: Determine mapping information according to sensor information.

[0080] The above mapping information is used to represent the mapping relationship between the current frame image block and the historical frame image block.

[0081] Image blocks can be obtained based on image frames.

[0082] For example, the current frame image block can be obtained based on the current frame image.

[0083] For example, the above historical frame image block can be obtained based on the current frame image.

[0084] In a possible implementation, the image frame may be divided into K image blocks, where K is a positive integer.

[0085] For example, K may be greater than 1 and smaller than M. M may satisfy: M=height of the image frame*width of the image block.

[0086] In one possible implementation, the historical frame image may be the N frames preceding the current frame image. N may be 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or another positive integer. For example, the historical frame image may be the one frame preceding the current frame image.

[0087] In one possible implementation, motion information can be determined based on the sensor information. The mapping information can be determined based on the motion information. The motion information includes a target translation amount and a target quaternion, where the target translation amount is the translation amount of the current frame image, and the target quaternion is the quaternion corresponding to the rotation matrix of the current frame image.

[0088] For example, the target angular velocity and target acceleration Determine the target translation and target quaternion. Determine the mapping information based on the target translation and target quaternion.

[0089] The above target translation Can satisfy:

[0090]

[0091] Among them, t represents the time, [Tk,Tk +1 ] represents the time period between the two frames of image acquisition time points of the camera. and They represent the IMU sampling time point bk corresponding to the current frame image (the current frame image is the next frame image relative to the previous frame image) +1 And the displacement relative to the world coordinate w at the sampling time point bk of the previous frame image, Δt represents the sampling frequency of IMU.

[0092] The above target quaternion Can satisfy:

[0093]

[0094] in, and They represent the IMU at the corresponding current frame image sampling time point bk +1 and the quaternion representation of the rotation angle relative to the world coordinate system w at the sampling time point bk of the previous frame image, The operator represents the multiplication of quaternions, and Ω represents quaternion operations.

[0095] According to the IMU, the sampling point bk corresponding to the current frame image +1 Displacement under Get the IMU translation matrix of the IMU pose, and calculate the IMU position according to the sampling point bk of the current frame image. +1 The quaternion of the rotation angle is used to obtain the IMU rotation matrix of the IMU pose, thereby obtaining the IMU pose in the world coordinate system.

[0096] Can satisfy:

[0097]

[0098] in, and They represent the IMU at the corresponding current frame image sampling time point bk +1 and the speed relative to the world coordinate system at the sampling time point bk of the previous frame image

[0099] In a possible implementation, the mapping information may be determined based on the motion information and intrinsic and extrinsic parameters of the camera (such as intrinsic and extrinsic parameters of an RGB camera).

[0100] It's easy to understand that using the RGB camera's internal and external parameters, we can project the position of each point in the previous frame onto the current frame. By calculating the projection relationship between the points within each image block (taking a weighted average of each point), we can find the mapping relationship between the image blocks of the previous and next frames. Image block mapping eliminates the problem of inaccurate individual pixel motion and is more robust than single-point mapping.

[0101] S203: Determine feature points of the current frame image block according to the mapping information.

[0102] The above-mentioned current frame image block is any image block in the above-mentioned current frame image.

[0103] For example, Figure 3 As shown, the above-mentioned current frame image block is the A1 image block in the above-mentioned current frame image.

[0104] It can be understood that in the previous frame image, an object is in image block A0. The camera moves, and in the image captured in the current frame, the object is in block A1. A1 and A0 are corresponding blocks, that is, image block A1 maps to image block A0 in the previous frame. The mapping relationship between the image blocks of the previous and current frames is obtained through the above IMU motion information and projection relationship.

[0105] In a possible implementation, the target image block of the current frame image block can be determined according to the above mapping information. The feature points of the current frame image block are determined according to the target information of the above target image block. Among them, the above target image block is the image block in the historical frame image corresponding to the current frame image block, and the above target information includes the number of feature points and the feature point extraction threshold.

[0106] Exemplarily, as Figure 3 shown, the target image block of the A1 image block in the current frame image can be determined as the A0 image block of the historical frame according to the above mapping information. The feature points of the A1 image block are determined according to the number of feature points of the A0 image block and the feature point extraction threshold of the A0 image block.

[0107] In a possible implementation, the feature point extraction threshold of the current frame image block can be determined according to the target information of the above target image block. The feature points of the current frame image block are determined according to the feature point extraction threshold of the current frame image block.

[0108] In a possible implementation, when the number of feature points of the above target image block is less than the first threshold, the feature point extraction threshold of the above target image block can be reduced to obtain the feature point extraction threshold of the current frame image block.

[0109] Exemplarily, when N’ < N_min, the feature point extraction threshold of the current frame image block can satisfy:

[0110] λ = λ' - Δλ

[0111] Where, λ is the feature point extraction threshold of the current frame image block, λ' is the feature point extraction threshold of the above target image block, and Δλ is the preset feature point adjustment threshold. N’ is the number of feature points of the above target image block, and N_min is the above first threshold.

[0112] It can be understood that if N’ < N_min, it means that the number of feature points of the target image block is small, then the adjustment threshold λ = Δλ is used to make the number of feature points extracted from the current frame image block larger.

[0113] In a possible implementation, when the number of feature points of the above target image block is greater than the second threshold, the feature point extraction threshold of the above target image block can be increased to obtain the feature point extraction threshold of the current frame image block.

[0114] For example, when N'>N_max, the feature point extraction threshold of the current frame image block may satisfy:

[0115] λ=λ'+Δλ

[0116] Where λ is the feature point extraction threshold of the current frame image block, λ' is the feature point extraction threshold of the target image block, Δλ is the preset feature point adjustment threshold, N' is the number of feature points of the target image block, and N_max is the second threshold.

[0117] It can be understood that if N'>N_max, it means that the target image block has a large number of feature points, and the threshold λ=λ'+Δλ is adjusted to reduce the number of feature points extracted from the current frame image block.

[0118] In a possible implementation, when the number of feature points of the target image block is greater than or equal to a first threshold and less than or equal to a second threshold, the feature point extraction threshold of the current frame image block can be determined according to the feature point extraction threshold of the target image block.

[0119] Exemplarily, when N_min≤N'≤N_max, the feature point extraction threshold of the current frame image block may satisfy:

[0120] λ=λ'

[0121] Where λ is the feature point extraction threshold for the current frame image block, λ' is the feature point extraction threshold for the target image block, and Δλ is the preset feature point adjustment threshold. N' is the number of feature points in the target image block, N_min is the first threshold, and N_max is the second threshold.

[0122] After determining the feature point extraction threshold of each image block, feature point extraction can be performed on each image block.

[0123] The specific method for extracting the above-mentioned feature points can adopt any method that can be thought of by those skilled in the art, and the embodiments of the present application are not limited to this.

[0124] For example, feature points of each image block may be extracted according to a feature from accelerated segment test (FAST) algorithm.

[0125] For another example, feature points of each image block may be extracted according to a speeded-up robust features (SURF) algorithm.

[0126] The following will be combined Figure 4An information processing device for executing the above information processing method is introduced.

[0127] It is understandable that, in order to realize the above functions, the information processing device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0128] In the embodiment of the present application, the information processing device can be divided into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0129] In the case of dividing each functional module into corresponding functional modules, Figure 4 A possible schematic diagram of the composition of the information processing device involved in the above embodiments is shown. The device can be an electronic device, or a module applied to an electronic device (such as a processor, chip or chip system, etc.), or a logical node, logical module or software that can realize all or part of the functions of the electronic device. Figure 4 As shown, the information processing device 400 may include: a transceiver unit 401 and a processing unit 402.

[0130] The transceiver unit 401 is used to obtain sensor information, where the sensor information includes target acceleration and target angular velocity. The target acceleration is the acceleration measured by the IMU, and the target angular velocity is the angular velocity measured by the IMU.

[0131] The processing unit 402 is configured to determine mapping information according to the sensor information, where the mapping information is used to represent a mapping relationship between the current frame image block and the historical frame image block.

[0132] The processing unit 402 is configured to determine feature points of a current frame image block according to the mapping information, where the current frame image block is any image block in the current frame image.

[0133] In one possible implementation, the processing unit is specifically used to: determine motion information based on the sensor information, the motion information including target translation and target quaternion, the target translation being the translation of the current frame image, and the target quaternion being the quaternion corresponding to the rotation matrix of the current frame image; and determine the mapping information based on the motion information.

[0134] In a possible implementation, the processing unit 402 is specifically used to: determine the target image block of the current frame image block based on the mapping information, where the target image block is the image block in the historical frame image that corresponds to the current frame image block; determine the feature points of the current frame image block based on the target information of the target image block, where the target information includes the number of feature points and the feature point extraction threshold.

[0135] In a possible implementation, the processing unit 402 is specifically used to: determine the feature point extraction threshold of the current frame image block according to the target information of the target image block; and determine the feature points of the current frame image block according to the feature point extraction threshold of the current frame image block.

[0136] In one possible implementation, the processing unit 402 is specifically configured to: when the number of feature points of the target image block is less than a first threshold, reduce the feature point extraction threshold of the target image block to obtain the feature point extraction threshold of the current frame image block; and when the number of feature points of the target image block is greater than a second threshold, increase the feature point extraction threshold of the target image block to obtain the feature point extraction threshold of the current frame image block.

[0137] An embodiment of the present application also provides a chip. Figure 5 FIG2 shows a schematic diagram of the structure of a chip 500. The chip 500 includes one or more processors 501 and an interface circuit 502. Optionally, the chip 500 may further include a bus 503.

[0138] The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above information processing method can be completed by the hardware integrated logic circuit in the processor 501 or the software instruction.

[0139] Optionally, the processor 501 may be a general-purpose processor, a digital signal processing (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods and steps disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor.

[0140] The interface circuit 502 can be used to send or receive data, instructions or information. The processor 501 can use the data, instructions or other information received by the interface circuit 502 to process it, and can send the processing completion information through the interface circuit 502.

[0141] Optionally, the chip also includes a memory, which may include a read-only memory and a random access memory, and provides operating instructions and data to the processor. Part of the memory may also include a non-volatile random access memory (NVRAM).

[0142] Optionally, the memory stores an executable software module or a data structure, and the processor can perform corresponding operations by calling an operation instruction stored in the memory (the operation instruction may be stored in an operating system).

[0143] Optionally, the chip can be used in an information processing device according to an embodiment of the present application. Optionally, the interface circuit 502 can be used to output the execution result of the processor 501. For information about the information processing method provided in one or more embodiments of the present application, reference can be made to the aforementioned embodiments and will not be repeated here.

[0144] It should be noted that the corresponding functions of the processor 501 and the interface circuit 502 can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0145] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 600 may be a processor or a chip or functional module in a processor. Figure 6 As shown, the electronic device 600 includes a processor 601 , a transceiver 602 and a communication circuit 603 .

[0146] Among them, the processor 601 is used to execute any step in the information processing method provided in the embodiment of the present application, and in the process of executing any step in the information processing method provided in the embodiment of the present application, the transceiver 602 and the communication line 603 can be optionally called to complete the corresponding operation.

[0147] Furthermore, the electronic device 600 may further include a memory 604 , wherein the processor 601 , the memory 604 and the transceiver 602 may be connected via a communication line 603 .

[0148] The processor 601 is a processor, a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 601 may also be other devices with processing functions, such as circuits, devices, or software modules, without limitation.

[0149] The transceiver 602 is used to communicate with other devices or other communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The transceiver 602 can be a module, a circuit, a transceiver, or any device capable of implementing communication.

[0150] The transceiver 602 is mainly used for sending and receiving commands and information, and may include a transmitter and a receiver for sending and receiving commands and information, respectively. Operations other than sending and receiving commands and information are implemented by the processor.

[0151] The communication line 603 is used to transmit information between the components included in the electronic device 600.

[0152] In one design, the processor can be considered as the logic circuit and the transceiver as the interface circuit.

[0153] The memory 604 is used to store instructions, where the instructions may be computer programs.

[0154] The memory 604 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM may be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). Memory 604 may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, etc. It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0155] It should be noted that the memory 604 can exist independently of the processor 601 or can be integrated with the processor 601. The memory 604 can be used to store instructions, program code, or some data. The memory 604 can be located within the electronic device 600 or outside the electronic device 600, without limitation. The processor 601 is configured to execute the instructions stored in the memory 604 to implement the methods provided in the above embodiments of the present application.

[0156] In one example, the processor 601 may include one or more processors, such as Figure 6 Processor 0 and processor 1 in.

[0157] As an optional implementation, the electronic device 600 includes multiple processors, for example, Figure 6 In addition to the processor 601, a processor 609 may also be included.

[0158] As an optional implementation, the electronic device 600 further includes an output device 605 and an input device 606. For example, the input device 606 is a keyboard, a mouse, a microphone, or a joystick, and the output device 605 is a display screen, a speaker, or the like.

[0159] It should be noted that the electronic device 600 can be a chip system or a Figure 6 Devices with similar structures in the chip system. Among them, the chip system can be composed of chips, or it can include chips and other discrete devices. The actions, terms, etc. involved in the various embodiments of this application can refer to each other without limitation. The message names or parameter names in the messages exchanged between the various devices in the embodiments of this application are only examples, and other names can also be used in specific implementations without limitation. In addition, Figure 6 The components shown in the figure do not constitute a limitation on the electronic device 600. Figure 6 In addition to the components shown, the electronic device 600 may include Figure 6 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0160] The processor and transceiver described in this application can be implemented on an integrated circuit (IC), an analog IC, a radio frequency integrated circuit, a mixed-signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), n-type metal oxide semiconductor (NMOS), p-type metal oxide semiconductor (positive channel metal oxide semiconductor, PMOS), bipolar junction transistor (BJT), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0161] Figure 7This is a schematic diagram of the structure of an information processing device provided in an embodiment of the present application. The information processing device can be applied to the scenario shown in the above method embodiment. For the convenience of explanation, Figure 7 Only the main components of the information processing device are shown, including a processor 701, a memory 702, a control circuit 703, and an input / output device 704. The processor 701 is primarily used to process communication protocols and communication data, execute software programs, and process software program data. The memory 702 is primarily used to store software programs and data. The control circuit 703 is primarily used to provide power and transmit various electrical signals. The input / output device 704 is primarily used to receive user input and output data to the user.

[0162] When the information processing device is a processor 701, the control circuit 703 may be a motherboard, the memory 702 includes a hard disk, RAM, ROM and other media with storage functions, the processor 701 may include a baseband processor 701 and a central processing unit, the baseband processor is mainly used to process communication protocols and communication data, the central processing unit is mainly used to control the entire information processing device, execute software programs, and process software program data, the input and output devices 704 include a display screen, a keyboard, and a mouse, etc.; the control circuit 703 may further include or be connected to a transceiver circuit or transceiver, such as a network cable interface, etc., for sending or receiving data or signals, such as for data transmission and communication with other devices. Furthermore, it may also include an antenna for sending and receiving wireless signals for data / signal transmission with other devices.

[0163] An embodiment of the present application further provides an information processing device, which includes: at least one processor, and when the at least one processor executes program code or instructions, it implements the above-mentioned related method steps to implement the information processing method in the above-mentioned embodiment.

[0164] Optionally, the apparatus may further include at least one memory configured to store the program code or instruction.

[0165] An embodiment of the present application also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an information processing device, the information processing device executes the above-mentioned related method steps to implement the information processing method in the above-mentioned embodiment.

[0166] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the information processing method in the above-mentioned embodiment.

[0167] The present application also provides an information processing device, which may be a chip, integrated circuit, component, or module. Specifically, the device may include a processor and a memory for storing instructions, or the device may include at least one processor for retrieving instructions from an external memory. When the device is running, the processor may execute the instructions, causing the chip to perform the information processing method described in each of the above method embodiments.

[0168] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0169] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of this embodiment.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0174] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the above methods of each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0175] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An information processing method, characterized in that: include: Acquire sensor information, where the sensor information includes a target acceleration and a target angular velocity, where the target acceleration is the acceleration measured by an inertial measurement unit (IMU), and the target angular velocity is the angular velocity measured by the IMU; Determining mapping information according to the sensor information, wherein the mapping information is used to represent a mapping relationship between a current frame image block and a historical frame image block; The feature points of a current frame image block are determined according to the mapping information, where the current frame image block is any image block in the current frame image.

2. The method according to claim 1, characterized in that The determining mapping information according to the sensor information includes: Determining motion information according to the sensor information, the motion information including a target translation amount and a target quaternion, the target translation amount being the translation amount of the current frame image, and the target quaternion being the quaternion corresponding to the rotation matrix of the current frame image; The mapping information is determined according to the motion information.

3. The method according to claim 1 or 2, characterized in that The determining the feature points of the current frame image block according to the mapping information includes: Determine a target image block of the current frame image block according to the mapping information, wherein the target image block is an image block in the historical frame image that corresponds to the current frame image block; The feature points of the current frame image block are determined according to target information of the target image block, where the target information includes the number of feature points and a feature point extraction threshold.

4. The method according to claim 3, characterized in that The determining the feature points of the current frame image block according to the target information of the target image block includes: Determining a feature point extraction threshold of the current frame image block according to target information of the target image block; The feature points of the current frame image block are determined according to a feature point extraction threshold of the current frame image block.

5. The method according to claim 4, characterized in that The step of determining a feature point extraction threshold of the current frame image block according to target information of the target image block includes: When the number of feature points of the target image block is less than a first threshold, reducing the feature point extraction threshold of the target image block to obtain the feature point extraction threshold of the current frame image block; When the number of feature points of the target image block is greater than the second threshold, the feature point extraction threshold of the target image block is increased to obtain the feature point extraction threshold of the current frame image block.

6. An information processing device, characterized in that include: transceiver unit and processing unit; The transceiver unit is used to obtain sensor information, wherein the sensor information includes a target acceleration and a target angular velocity, wherein the target acceleration is the acceleration measured by the IMU, and the target angular velocity is the angular velocity measured by the IMU; The processing unit is configured to determine mapping information based on the sensor information, wherein the mapping information is used to represent a mapping relationship between a current frame image block and a historical frame image block; The processing unit is configured to determine feature points of a current frame image block according to the mapping information, where the current frame image block is any image block in the current frame image.

7. The device according to claim 6, characterized in that The processing unit is specifically configured to: Determining motion information according to the sensor information, the motion information including a target translation amount and a target quaternion, the target translation amount being the translation amount of the current frame image, and the target quaternion being the quaternion corresponding to the rotation matrix of the current frame image; The mapping information is determined according to the motion information.

8. The device according to claim 6 or 7, characterized in that The processing unit is specifically configured to: Determine a target image block of the current frame image block according to the mapping information, wherein the target image block is an image block in the historical frame image that corresponds to the current frame image block; The feature points of the current frame image block are determined according to target information of the target image block, where the target information includes the number of feature points and a feature point extraction threshold.

9. The device according to claim 8, characterized in that The processing unit is specifically configured to: Determining a feature point extraction threshold of the current frame image block according to target information of the target image block; The feature points of the current frame image block are determined according to a feature point extraction threshold of the current frame image block.

10. The device according to claim 9, characterized in that The processing unit is specifically configured to: When the number of feature points of the target image block is less than a first threshold, reducing the feature point extraction threshold of the target image block to obtain the feature point extraction threshold of the current frame image block; When the number of feature points of the target image block is greater than the second threshold, the feature point extraction threshold of the target image block is increased to obtain the feature point extraction threshold of the current frame image block.

11. An information processing device comprising at least one processor and a memory, characterized in that: The at least one processor executes a program or instruction stored in the memory, so that the information processing device implements the method according to any one of claims 1 to 5.

12. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed on a computer or a processor, the computer or the processor is caused to implement the method according to any one of claims 1 to 5.

13. A computer program product comprising instructions, characterized in that: When the instructions are executed on a computer or a processor, the computer or the processor is caused to implement the method according to any one of claims 1 to 5 .

14. A chip comprising at least one processor and a memory, characterized in that: The at least one processor executes a program or instruction stored in the memory to implement the method according to any one of claims 1 to 5.

Images