Image processing method and device

Abstract

The invention provides an image processing method and device which are used for improving image processing effects (such as image registration and target recognition). The method comprises the following steps: acquiring RGB values and depth values of N frames of images; according to the RGB value and the depth value of the ith frame of image in the N frames of images, layer information of the ith frame of image is determined, and the layer information indicates a layer included in the ith frame of image; and carrying out image processing on the N frames of images based on the N pieces of image layer information corresponding to the N frames of images.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method and device. Background Art

[0002] With the development of image processing technology, image processing technology has been widely used in smart home, smart car and other fields. Image signal processor (ISP) is a processor set in the image processing device. ISP can process the image signal output by the front-end image sensor to realize image registration, video anti-shake, target recognition and other applications.

[0003] However, ISP is limited by reasons such as fast camera motion, depth changes in images, and fast changes in color, brightness, or texture of images, and there are bottlenecks in image processing effects in applications such as video stabilization, image registration, and target recognition. For example, in image registration applications, ISP usually extracts feature information from RGB images and calculates the optical flow of the image based on the features to achieve image registration; however, weak textures or repeated textures in images will cause image extraction failures, and brightness changes between frames will cause errors in optical flow calculations, which will lead to image registration failures and generate image noise.

[0004] Therefore, how to improve the image processing effect of ISP is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0005] The present application provides an image processing method and device to improve the effect of image processing.

[0006] In a first aspect, an embodiment of the present application provides an image processing device, comprising: an acquisition unit, configured to acquire RGB values ​​of N frames of images; and, to acquire depth values ​​of the N frames of images; wherein N is an integer greater than or equal to 1; a processing unit, configured to determine layer information of the i-th frame of image based on the RGB value of the i-th frame of image and the depth value of the i-th frame of image in the N frames of image; wherein i is any positive integer between 1 and N, and the layer information indicates the layer included in the i-th frame of image; the processing unit is further configured to perform image processing on the N frames of image based on the N layer information corresponding to the N frames of image.

[0007] In the embodiment of the present application, the layer included in the i-th frame image may be at least one of a foreground layer, a background layer, or a mid-ground layer. The mid-ground layer includes one or more intermediate layers other than the foreground layer and the background layer obtained when the i-th frame image is subjected to scene segmentation.

[0008] The image processing device can determine the layer information of the i-th frame image in combination with the RGB value of the i-th frame image and the depth value of the i-th frame image, and perform image processing on the N-frame image based on the N layer information corresponding to the N-frame image, which can effectively improve the image processing effect (such as image registration, image background blur, or target recognition and other effects). For example, when the layer information in the embodiment of the present application is used for image registration, the image processing device can successfully extract feature points in the N-frame image according to the layer information, thereby achieving effective image registration. For another example, when the layer information in the embodiment of the present application is used for background blur, the image processing device can perform accurate background blur according to the layer information, thereby effectively improving the background blur effect. For another example, when the layer information in the embodiment of the present application is used for target recognition, since different targets are segmented in different layers, the accuracy of target recognition can be effectively improved.

[0009] In a possible design, the image processing device also includes a receiving unit; the receiving unit is used to receive the echo signal corresponding to the i-th frame image; the processing unit is also used to obtain the number of echo photons corresponding to each pixel in the i-th frame image according to the echo signal corresponding to the i-th frame image, and generate the confidence of each pixel; and, according to the confidence of each pixel, determine the confidence map of the i-th frame image, and the confidence map is used to reflect the credibility of the depth value of each pixel. In this design, the image processing device can generate a confidence map of the i-th frame image, so that when the i-th frame image is subsequently processed according to the confidence map, the image processing effect can be improved. For example, when the image processing device performs target recognition on the i-th frame image based on the confidence map, the accuracy of target recognition can be effectively improved.

[0010] In a possible design, the image processing device also includes a receiving unit; the receiving unit is used to receive the echo signal corresponding to the i-th frame image; the processing unit is also used to obtain the number of echo photons corresponding to all pixels in the i-th frame image according to the echo signal corresponding to the i-th frame image; based on the number of echo photons corresponding to all pixels, an intensity map of the i-th frame image is generated, and the intensity map is used to reflect the contour and / or edge of the target object in the i-th frame image. In this design, the image processing device can generate an intensity map of the i-th frame image, so that when the i-th frame image is subsequently processed according to the intensity map, the image processing effect can be improved. For example, based on the intensity map, when the i-th frame image is subjected to target recognition, the accuracy of target recognition can be effectively improved.

[0011] In a possible design, the layer information corresponding to two adjacent frames of images in N frames of images is used to determine the motion information between the two adjacent frames of images; wherein the motion information includes global motion information and / or local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the two adjacent frames of images, and the local motion information is used to characterize the motion state of the target object in the two adjacent frames of images. In this design, when the motion information is used for image processing, it can effectively improve the image processing effect (such as image registration, video stabilization, etc.). For example, when the motion information is used for video stabilization, it can effectively describe the real motion of the image acquisition device in three-dimensional space, thereby effectively improving the video stabilization effect in scenes where the image acquisition device moves faster. For another example, when the motion information is used for video stabilization, it can effectively describe the motion state of the target object in the image in three-dimensional space, thereby effectively improving the video stabilization effect in scenes where the depth of the object in the image changes.

[0012] For example, the image processing device can determine the layer information of the i+1th frame image based on the RGB value of the i+1th frame image and the depth value of the i+1th frame image; determine the motion information between the i-frame image and the i+1th frame image based on the layer information of the i-frame image and the layer information of the i+1th frame image; wherein the motion information includes global motion information and local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the i-frame image and the i+1th frame image, and the local motion information is used to characterize the motion state of the target object in the i-frame image and the i+1th frame image.

[0013] Optionally, the layer information of the i-th frame image is determined according to histogram A, which is used to characterize the depth values ​​of each pixel value in the i-th frame image, and histogram A includes M depth levels; the layer information of the i+1-th frame image is determined according to histogram B, which is used to characterize the depth values ​​of each pixel value in the i+1-th frame image, and histogram B includes M depth levels; accordingly, according to the layer information of the i-th frame image and the layer information of the i+1-th frame image, the motion information between the i-th frame image and the i+1-th frame image is determined, including: comparing the M depth levels included in the histogram A with the M depth levels included in the histogram B, and if the difference between the two is less than a third threshold, it is considered that the global motion state from the i-th frame image to the i+1-th frame image is static state, otherwise, the global motion state is considered to be a motion state; similarly, according to the layer information of the i-th frame image, the region of interest ROI of the i-th frame image is determined; according to the layer information of the i+1-th frame image, the ROI of the i+1-th frame image is determined; and, the first N depth levels with higher depth values ​​of the ROI of the i-th frame image in the histogram A are determined, and the first N depth levels with higher depth values ​​of the ROI of the i+1-th frame image in the histogram B are determined, and the first N depth levels included in the histogram A are compared with the first N depth levels included in the histogram B. If the difference between the two is less than the fourth threshold, it is considered that the local motion state from the i-th frame image to the i+1-th frame image is a static state, otherwise, the local motion state is considered to be a motion state.

[0014] In one possible design, the processing unit is further used to: determine 3D reprojection information between two adjacent frames of images; wherein the 3D reprojection information is used to screen the target feature point pairs between two adjacent frames of images. In this design, the 3D reprojection information between adjacent frames can be determined, and when the 3D reprojection information is applied to video stabilization, it can effectively describe the motion state of the target object in the image in the three-dimensional space, thereby effectively improving the effect of video stabilization.

[0015] For example, the processing unit can transform the 2D feature point pixel coordinates of the i-th frame image in N frames to obtain a first 3D spatial coordinate; transform the 2D feature point pixel coordinates of the i+1-th frame image in N frames to obtain a second 3D spatial coordinate; determine the 3D reprojection information between the i-th frame image and the i+1-th frame image based on the first 3D spatial coordinate and the second 3D spatial coordinate, and the 3D reprojection information is used to screen the target feature point pairs between the i-th frame image and the i+1-th frame image.

[0016] In one possible design, the apparatus further includes a sending unit, which is used to send the 3D reprojection information to the image processing device. In this design, the image processing apparatus can send the 3D reprojection information to other image processing devices, so that the other image processing devices can perform corresponding image processing, such as implementing applications such as video stabilization based on the 3D reprojection information.

[0017] In one possible design, N layers of information are used to determine the region of interest ROI in N frames of images, and the first feature point in the ROI is used for image registration. In this design, the electronic device can accurately extract the first feature point in the ROI in the N frames of images based on the N layers of information, and then can achieve effective image registration based on the first feature point, thereby effectively improving the effect of image registration.

[0018] In one possible design, the sending unit is further used to send N layers of information to the image processing device. In this design, the image processing device can also send the N layers of information to other image processing devices for the other image processing devices to perform image processing (such as image background blur, target recognition, etc.).

[0019] In a possible design, the coordinate system where the RGB value of the i-th frame image is located is the first coordinate system, and the depth value of each pixel in the i-th frame image in the first coordinate system includes M depth levels; when M=2, the layer information indicates that the layer of the i-th frame image includes a foreground layer and a background layer; when M>2, the layer information indicates that the layer of the i-th frame image includes a foreground layer, a midground layer, and a background layer. Among them, the midground layer may include one or more intermediate layers, and an intermediate layer may be, for example, a layer where an object in the image is located. Optionally, one of the M depth levels corresponds to one of the first M depth values ​​with the largest count value in the second histogram. In this design, for images with more complex depth values, the image can be segmented into at least one of the foreground layer, the midground layer, or the background layer according to the depth levels in the image, so that the scene segmentation accuracy of the image is higher, thereby improving the image processing effect of complex images.

[0020] In a possible design, the processing unit is further used to: when the H angles between the first spatial normal vector and the second spatial normal vector are greater than a first threshold, interpolate and correct the depth value of the second pixel according to the depth values ​​of the adjacent pixels of the second pixel; wherein the first spatial normal vector is the first spatial normal vector between the second pixel in the i-th frame image and the image acquisition device of the i-th frame image, the second spatial normal vector includes G spatial normal vectors between the adjacent pixels of the second pixel and the image acquisition device, H is an integer greater than or equal to 1, and G is an integer greater than or equal to H. In this design, the depth values ​​of some pixels with inaccurate depth values ​​can be interpolated and corrected, and the error influence of scenes such as strong reflectivity and strong ambient light on the depth value of the i-th frame image can be eliminated, thereby further improving the accuracy of the depth value of the i-th frame image.

[0021] In one possible design, the acquisition unit is a direct time of flight (DTOF) camera, and the depth value of N frames of images can be acquired by the DTOF camera. In this design, the accuracy of the depth value of N frames of images acquired by the DTOF camera is high, which can effectively improve the accuracy of scene segmentation, and further improve the effect of subsequent image processing. For example, when the layer information is used for background blur, the background blur effect can be further improved.

[0022] In a possible design, the i-th frame image includes J pixels, and the depth value of the i-th frame image includes the depth values ​​of J pixels; accordingly, the depth value of the N-frame image is obtained by the DTOF camera, including: receiving the first echo data from the DTOF camera, the first echo data includes J first histograms, and the j-th first histogram in the J first histograms is used to characterize the echo position corresponding to the j-th pixel in the i-th frame image; wherein j is any positive integer between 1 and J; when there are multiple echo positions of the j-th pixel in the j-th first histogram, the frequency of each echo position in the echo position of the first pixel is counted, and the echo position with the highest frequency in the multiple echo positions is used as the echo position corresponding to the j-th pixel; wherein the first pixel includes multiple pixels adjacent to the j-th pixel in the spatial domain of the j-th first histogram; and according to the echo position corresponding to the j-th pixel, the depth value of the j-th pixel is determined. wherein the J first histograms are generated according to the flight time of each laser emitted by the laser sensor in the DTOF camera. In this design, the accuracy of the echo position of the pixel can be effectively improved, so that the depth value of the pixel is more accurate, and thus the layer information subsequently determined based on the depth value of the pixel is more accurate.

[0023] In a possible design, when the i-th frame image includes J pixels, the depth value of the i-th frame image includes the depth values ​​of J pixels, the depth values ​​of J pixels correspond to the coordinate system where the DTOF camera is located, the coordinate system where the DTOF camera is located is the second coordinate system, and the coordinate system where the RGB value of the i-th frame image is located is the first coordinate system; accordingly, the process of the electronic device determining the layer information of the i-th frame image according to the RGB value and depth value of the i-th frame image in the N-frame image can be: converting the depth value corresponding to the j-th pixel in the i-th frame image from the second coordinate system to the first coordinate system to obtain the first RGBD information, and the first RGBD information includes the depth value of each pixel in the i-th frame image in the first coordinate system. Wherein, j takes any positive integer between 1 and J; according to the first RGBD information, the layer information of the i-th frame image is determined. In this design, the first RGBD information is obtained by aligning the RGB value and the depth value of the i-th frame image, so that the layer information of the i-th frame image is determined more accurately according to the first RGBD information.

[0024] Optionally, the process of converting the depth value of the jth pixel in the i-th frame image from the second coordinate system to the first coordinate system by the electronic device can be: obtaining an intensity map of the first scene through a TOF camera, and obtaining an RGB map of the first scene through an RGB camera, calibrating the external parameters of the DTOF camera and the RGB camera according to the intensity map and the RGB map, and obtaining the rotation matrix and translation matrix between the DTOF camera and the RGB camera; according to the rotation matrix and the translation matrix, converting the depth value corresponding to the jth pixel in the i-th frame image from the second coordinate system where the DTOF camera is located to the first coordinate system where the RGB camera is located, and obtaining the first RGBD information. Among them, the first scene can be, for example, a checkerboard scene, and the embodiments of the present application are not specifically limited. Among them, the external parameters of the RGB camera refer to the parameters used in the process of converting the RGB value from the world coordinate system to the camera coordinate system, and the external parameters of the DTOF camera refer to the parameters used in the process of converting the depth value from the world coordinate system to the camera coordinate system.

[0025] Optionally, after obtaining the first RGBD information, the processing unit may also interpolate the depth value of the second pixel according to the depth value of the third pixel in the i-th frame image when the depth value of the second pixel in the i-th frame image is missing, to obtain second RGBD information; wherein the third pixel is adjacent to the second pixel, and the confidence corresponding to the depth value of the third pixel is greater than the second threshold; and determine the layer information of the i-th frame image according to the second RGBD information. In this design, the second RGBD information can be obtained by interpolating the pixels with missing depth values, thereby making the layer information determined based on the second RGBD information more accurate.

[0026] In a second aspect, an embodiment of the present application further provides an image processing method, which can be applied to an electronic device or a server, and the method includes: obtaining RGB values ​​of N frames of images; wherein N is an integer greater than or equal to 1; taking the depth value of the N frames of images; determining the layer included in the i-th frame of image according to the RGB value of the i-th frame of image in the N frames of image and the depth value of the i-th frame of image; wherein i is any positive integer between 1 and N; the layer information indicates the layer included in the i-th frame of image; and performing image processing on the N frames of image based on the N layer information corresponding to the N frames of image.

[0027] In one possible design, the method also includes: receiving an echo signal corresponding to the i-th frame image; obtaining the number of echo photons corresponding to each pixel in the i-th frame image according to the echo signal corresponding to the i-th frame image, and generating the confidence of each pixel; determining a confidence map of the i-th frame image according to the confidence of each pixel, wherein the confidence map is used to reflect the credibility of the depth value of each pixel.

[0028] In one possible design, the method also includes: receiving an echo signal corresponding to the i-th frame image; obtaining the number of echo photons corresponding to all pixels in the i-th frame image based on the echo signal corresponding to the i-th frame image; generating an intensity map of the i-th frame image based on the number of echo photons corresponding to all pixels, the intensity map being used to reflect the contour and / or edge of the target object in the i-th frame image.

[0029] In a possible design, the layer information corresponding to two adjacent frames of the N frames of images is used to determine the motion information between the two adjacent frames of images; wherein the motion information includes global motion information and / or local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the two adjacent frames of images, and the local motion information is used to characterize the motion state of the target object in the two adjacent frames of images.

[0030] In one possible design, the method further includes: determining 3D reprojection information between two adjacent frames of images; wherein the 3D reprojection information is used to screen target feature point pairs between the two adjacent frames of images.

[0031] In a possible design, the N layer information is used to determine a region of interest ROI in the N frames of images, and the first feature point in the ROI is used for image registration.

[0032] In a possible design, the coordinate system where the RGB values ​​of the i-th frame image are located is a first coordinate system, and the depth values ​​of each pixel in the i-th frame image in the first coordinate system include M depth levels; when M=2, the layer information indicates that the layers of the i-th frame image include a foreground layer and a background layer; when M>2, the layer information indicates that the layers of the i-th frame image include a foreground layer, a midground layer and a background layer.

[0033] In a possible design, the method further includes: when H angles between the first spatial normal vector and the second spatial normal vector are greater than a first threshold, interpolation correction is performed on the depth value of the second pixel according to the depth values ​​of the adjacent pixels of the second pixel; wherein the first spatial normal vector is the first spatial normal vector between the second pixel in the i-th frame image and the image acquisition device of the i-th frame image, the second spatial normal vector includes G spatial normal vectors between the adjacent pixels of the second pixel and the image acquisition device, the H is an integer greater than or equal to 1, and the G is an integer greater than or equal to the H. In this design, the depth values ​​of some pixels with inaccurate depth values ​​can be interpolated and corrected, and the error influence of scenes such as strong reflectivity and strong ambient light on the depth value of the i-th frame image can be eliminated, thereby further improving the accuracy of the depth value of the i-th frame image.

[0034] In a possible design, obtaining depth values ​​of N frames of images includes: obtaining depth values ​​of N frames of images by directly measuring a time-of-flight (DTOF) camera.

[0035] In a possible design, the i-th frame image includes J pixels, and the depth value of the i-th frame image includes the depth values ​​of the J pixels; accordingly, the depth value of the N-frame image is obtained through the DTOF camera, including: receiving the first echo data from the DTOF camera, the first echo data includes J first histograms, and the j-th first histogram in the J first histograms is used to characterize the echo position corresponding to the j-th pixel in the i-th frame image; wherein, the j takes any positive integer between 1 and the J; when there are multiple echo positions of the j-th pixel in the j-th first histogram, the frequency of each echo position in the multiple echo positions in the echo position of the first pixel is counted, and the echo position with the highest frequency among the multiple echo positions is used as the echo position corresponding to the j-th pixel; wherein, the first pixel includes multiple pixels adjacent to the j-th pixel in the spatial domain of the j-th first histogram; and the depth value of the j-th pixel is determined according to the echo position corresponding to the j-th pixel. Among them, the J first histograms are generated according to the flight time of each laser emitted by the laser sensor in the DTOF camera.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors, a memory, and one or more computer programs; wherein the computer program is stored in the memory, and the computer program is called when the processor is running, so that the electronic device executes the method described in the second aspect and any possible design of the second aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a server, comprising one or more processors, a memory, and a computer program; wherein the computer program is stored in the memory; when the processor is running, the computer program is called, so that the server executes the method described in the second aspect and any possible design of the second aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a chip, which is coupled to a memory in an electronic device so that the chip calls a computer program stored in the memory during operation to implement a method as described in the second aspect and any possible design in the second aspect.

[0039] In the sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program runs on the electronic device, the electronic device executes the method described in the second aspect and any possible design of the second aspect.

[0040] In a seventh aspect, the present application provides a computer program product, which, when running on a processor, implements the method described in any one of the designs in the second aspect above.

[0041] In an eighth aspect, the present application provides an image processing device, comprising a processor and a memory, wherein the memory stores a program code, and the processor is used to call the program code to execute the method as described in the second aspect and any one of the possible designs of the second aspect.

[0042] For the beneficial effects of the second to eighth aspects mentioned above, please refer to the technical effects that can be achieved by the corresponding design in the first aspect mentioned above, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A possible system architecture diagram provided for an embodiment of the present application;

[0044] Figure 2 One of the scene schematic diagrams of the image processing method provided in the embodiment of the present application;

[0045] Figure 3The second scene diagram of the image processing method provided in the embodiment of the present application;

[0046] Figure 4 One of the flowcharts of an image processing method provided in an embodiment of the present application;

[0047] Figure 5 The following is a schematic diagram showing the structure of an image processing system provided by an embodiment of the present application;

[0048] Figure 6 The following is a schematic diagram showing the operation flow of the DTOF preprocessing module provided in the embodiment of the present application;

[0049] Figure 7 One of the operation flow diagrams of the dual-camera fusion processing module provided in an embodiment of the present application is exemplarily shown;

[0050] Figure 8 The second schematic diagram of the operation flow of the dual-camera fusion processing module provided in the embodiment of the present application is exemplarily shown;

[0051] Fig. 9 The third schematic diagram of the operation flow of the dual-camera fusion processing module provided in the embodiment of the present application is exemplarily shown;

[0052] Fig.10 A schematic diagram of a video stabilization process provided by an embodiment of the present application is exemplarily shown;

[0053] Fig.11 A schematic diagram showing an exemplary process of image registration provided in an embodiment of the present application;

[0054] Fig.12 One of the structural schematic diagrams of an image processing device provided in an embodiment of the present application;

[0055] Fig.13 A second structural diagram of an image processing device provided in an embodiment of the present application;

[0056] Fig.14 The third structural schematic diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] To facilitate understanding, technical terms involved in the embodiments of the present application are introduced accordingly.

[0058] 1. Image registration refers to the process of matching and superimposing two or more frames of images acquired at different times, with different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.). The process includes determining the matching feature point pairs in the two frames of images, and then determining the image space coordinate transformation parameters based on the matching feature point pairs; finally, image registration is performed based on the image space coordinate transformation parameters.

[0059] 2. Direct time of flight (DTOF) is a new solution for obtaining depth values. In this solution, the DTOF camera uses a high-precision circuit delay measurement device to measure the flight time of the laser emitted by the pulsed laser transmitter in the target scene, and then converts it into the corresponding depth based on the speed of light.

[0060] The specific working principle of the DTOF camera includes: the laser transmitter in the DTOF camera emits a pulsed laser of a specific wavelength to the target scene and records the emission time; the obstacles in the target scene reflect the corresponding echo signal after receiving the pulsed laser, and the Single Photon Avalanche Diode (SPAD) in the DTOF camera detects the echo signal and records the arrival time of the echo photon in the echo signal; the processing module in the DTOF camera determines the time difference between the arrival time of the echo photon detected by the SPAD and the time of the pulse emission closest to it, and records the time difference as one detection; the DTOF camera repeats multiple detections (usually up to thousands of times), and counts the time difference each time as a histogram; the processing module in the DTOF camera parses the histogram to obtain the corresponding depth value.

[0061] Compared with traditional depth acquisition solutions (such as binocular vision, structured light, indirect time of flight (ITOF) signals, etc.), DTOF can obtain the depth values ​​of objects at a farther distance in the target scene (for example, vehicle-mounted DTOF can obtain the depth values ​​of target objects hundreds of meters away), and has the advantages of high detection accuracy (up to millimeter level), strong anti-interference ability (that is, stable measurement accuracy), low power consumption, and strong resistance to multipath crosstalk.

[0062] 3. Image Signal Processing (ISP), which can process the image signal output by the front-end image sensor to achieve applications such as image registration, video anti-shake, and target recognition. In the embodiment of the present application, the ISP can be set in an electronic device or a dedicated image processing device, and the embodiment of the present application does not make specific restrictions.

[0063] 4. RGB-D camera can obtain the RGB value and depth value of the image. In one possible implementation, the RGB-D camera can obtain the depth value of the image through infrared structured light (Structured Light), but this implementation is limited by lighting conditions and object materials. In another possible implementation, the RGB-D camera can obtain the depth value of the image through Time of Flight (TOF), and the depth value obtained by this implementation is more reliable.

[0064] The terms "system" and "network" in the embodiments of the present application can be used interchangeably. "At least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0065] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the priority or importance of multiple objects. For example, the first 2D coordinate and the second 2D coordinate are only used to distinguish different 2D coordinates, and do not indicate the difference in priority or importance of the two 2D coordinates.

[0066] As described above, the ISP is limited by the fast camera motion, depth changes in the image, color, brightness, or texture of the image, etc., and there is a bottleneck in the image processing effect in applications such as video stabilization, image registration, and target recognition.

[0067] For example, in image registration applications, ISP usually extracts feature information of three primary colors (Red Green Blue, RGB) images and calculates the optical flow of the image based on the features to achieve image registration; however, weak texture or repeated texture in the image will cause image extraction failure, and brightness changes between frames will cause optical flow calculation errors, resulting in image registration failure and further image noise.

[0068] For example, in video stabilization applications, ISP performs video stabilization based on the feature information and trajectory posture obtained from RGB images. Motion perception and trajectory smoothing are performed only based on the feature information in a two-dimensional plane. However, the feature information and trajectory posture are difficult to effectively describe the real motion of the camera in three-dimensional space, resulting in limited video stabilization effects in special scenarios such as scenes with fast camera motion and scenes with large changes in the depth of objects in the image.

[0069] In view of this, the present application provides an image processing method and an image processing device to improve the image processing effect. The image processing device can determine the layer information of the i-th frame image by combining the RGB value of the i-th frame image and the depth value of the i-th frame image, and perform image processing on the N-frame image based on the N layer information corresponding to the N-frame image, which can effectively improve the image processing effect.

[0070] For example, when the layer information in the embodiment of the present application is used for image registration, the image processing device can successfully extract feature points in N frames of images based on the layer information, thereby achieving effective image registration. For another example, when the layer information in the embodiment of the present application is used for background blurring, the image processing device can perform accurate background blurring based on the layer information, thereby effectively improving the background blurring effect. For another example, when the layer information in the embodiment of the present application is used for target recognition, since different targets are segmented into different layers, the accuracy of target recognition can be effectively improved.

[0071] The image processing method in the embodiment of the present application can be applied to an image processing system, which can be directly deployed on an electronic device or on a server. For example, it can be applied to a vehicle with an image processing function, or other devices with an image processing function in a vehicle. The other devices include but are not limited to: other sensors such as a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, a vehicle-mounted unit, a vehicle-mounted radar or a vehicle-mounted camera. The vehicle can implement the image processing method provided by the present application through the vehicle-mounted terminal, the vehicle-mounted controller, the vehicle-mounted module, the vehicle-mounted module, the vehicle-mounted component, the vehicle-mounted chip, the vehicle-mounted unit, the vehicle-mounted radar or the vehicle-mounted camera. Of course, the image processing scheme in the embodiment of the present application can also be used in other intelligent terminals with image processing functions other than the vehicle, or be set in other intelligent terminals with image processing functions other than the vehicle, or be set in the components of the intelligent terminal. The intelligent terminal can be an intelligent transportation device, an intelligent home device, a robot, etc. For example, it includes but is not limited to other sensors such as a controller, a chip, a radar or a camera in an intelligent terminal or an intelligent terminal, and other components.

[0072] The image processing results obtained in the embodiments of the present application can be applied to the fields of smart terminals, vehicle navigation, vehicle AR-HUD display, mobile phone AR navigation, robot navigation, AR digital content display, etc., and the embodiments of the present application are not specifically limited. For example, the image processing method provided in the embodiments of the present application can be applied to AR-related applications including AR navigation, AR games, store enhancement, city check-in, digital map crows, etc., and can also be deployed on robots and applied to the positioning of inspection robots, drones, AGVs and other robots.

[0073] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0074] First, the system architecture applicable to the embodiments of the present application is introduced in detail.

[0075] Figure 1 A possible system architecture diagram applicable to the embodiment of the present application is shown in FIG. Figure 1 The system architecture shown includes a collection device 110 , a server 120 , and an electronic device 130 .

[0076] The acquisition device 110 refers to a device with an image acquisition function, such as an RGB-D camera, a DTOF camera, or an RGB camera. Exemplarily, when the acquisition device 110 is applied to the field of mobile terminals, the acquisition device 110 may be, for example, a mobile phone, a computer, a tablet, etc. The acquisition device 110 is optional. When the electronic device 130 has an image acquisition function, the acquisition device may not be set in the system.

[0077] The server 120 may refer to a single server or a server cluster consisting of multiple servers. The server 120 may specifically be a cloud server, also known as a cloud, cloud, cloud server, cloud controller or Internet of Vehicles server. A cloud server is a general term for devices or devices with data processing capabilities, such as physical devices such as a host or processor (e.g., ISP), virtual devices such as a virtual machine or container, and chips or integrated circuits.

[0078] The electronic device 130 may be a mobile terminal with image processing function (for example, a mobile phone, a computer, a tablet, etc.). The electronic device 130 may also be registered on the server 120 to obtain image processing services provided by the server 120, such as image registration service, video anti-shake service, image background blur service, etc.

[0079] It should be understood that the embodiments of the present application do not limit the types and quantities of acquisition devices, the number of servers, and the number of electronic devices in the system architecture. For example, one server can only interact with one acquisition device and one electronic device, or it can interact with multiple acquisition devices and one electronic device, or it can interact with multiple acquisition devices and multiple electronic devices. The information interaction between two devices can refer to the direct interaction between the two devices through a wired or wireless method, or it can refer to the indirect interaction between the two devices through one or more other devices. Moreover, in addition to the acquisition devices, servers, and electronic devices, the system architecture to which the embodiments of the present application are applicable can also include other devices, such as terminal devices, network devices, core network devices, etc., which are not limited by the embodiments of the present application. In addition, any device in the embodiments of the present application can integrate functions into an independent physical unit, or distribute functions on multiple independent physical units, which are not limited by the embodiments of the present application.

[0080] based on Figure 1 The system architecture shown in the figure, the image processing method provided in the embodiment of the present application includes but is not limited to the following implementation methods:

[0081] In implementation mode 1, the electronic device 130 does not have an image acquisition function, and the acquisition device 110 obtains RGB values ​​and depth values ​​of N frames of images, and uploads the RGB values ​​and depth values ​​of the N frames of images to the server 120; then the server 120 determines N layers of information corresponding to the N frames of images based on the RGB values ​​and depth values ​​of the N frames of images; when the electronic device 130 has an image processing function, then the electronic device 130 can obtain N layers of information from the server 120, and perform image processing based on the N layers of information.

[0082] In embodiment 2, the electronic device 130 has image acquisition function and image processing function. Figure 2 As shown, when the electronic device 130 receives an image processing service request, the electronic device 130 collects the RGB values ​​and depth values ​​of N frames of images, and fuses the RGB values ​​and depth values ​​of the N frames of images to obtain layer information of the N frames of images, and then performs image processing based on the N layer information, and outputs the corresponding image processing results.

[0083] In implementation mode 3, the server 120 has an image processing function. Figure 3As shown, when the acquisition device 110 receives an image processing service request, the acquisition device 110 acquires RGB values ​​and depth values ​​of N frames of images, and uploads the RGB values ​​and depth values ​​of the N frames of images to the server 120; then the server 120 determines N layer information corresponding to the N frames of images according to the RGB values ​​and depth values ​​of the N frames of images, and sends the N layer information to the acquisition device 110, and the acquisition device 110 performs image processing based on the N layer information.

[0084] The above introduces the system architecture applicable to the embodiments of the present application. The following describes the image processing method provided by the embodiments of the present application in conjunction with specific drawings.

[0085] Figure 4 A schematic diagram showing a flow chart corresponding to an image processing method provided in an embodiment of the present application is exemplarily shown. The method is applicable to an electronic device having an image processing function (for example, a Figure 1 Any one of the collection device 110, server 120, or electronic device 130 shown in FIG. Figure 4 As shown, the method includes:

[0086] S401, obtaining RGB values ​​of N frames of images.

[0087] In a possible implementation, the electronic device has an image acquisition function, and the electronic device acquires RGB values ​​of N frames of images.

[0088] In another possible implementation, the electronic device does not have an image acquisition function, and the electronic device sends an acquisition instruction to the acquisition device to instruct the acquisition device to acquire RGB values ​​of N frames of images in its usage scenario; and the electronic device receives the RGB values ​​of N frames of images from the acquisition device.

[0089] Specifically, the electronic device or the acquisition device may acquire RGB values ​​of N frames of images through an RGB camera.

[0090] S402, obtaining depth values ​​of N frames of images.

[0091] In one possible implementation, the electronic device may obtain the depth value of N frames of images through a DTOF camera. In another possible implementation, the electronic device may obtain the depth value of N frames of images through an RGB-D camera in a DTOF manner. In this way, the depth values ​​of the N frames of images obtained have a high accuracy, which can effectively improve the scene segmentation accuracy of the N frames of images, and thus improve the effect of subsequent image processing.

[0092] S403 , determining layer information of the i-th frame image according to the RGB value of the i-th frame image and the depth value of the i-th frame image among the N frames of images.

[0093] The layer information indicates the layer included in the i-th frame image. The i-th frame image may include at least one of a foreground layer, a background layer, or a mid-ground layer. The mid-ground layer may include one or more intermediate layers, and an intermediate layer may be, for example, a layer where a target object in the image is located.

[0094] In S403, the process of the electronic device determining the layer information of the i-th frame image according to the RGB value of the i-th frame image and the depth value of the i-th frame image in N frames of images may be: converting the depth value of the i-th frame image to the coordinate system where the RGB value is located, and determining the layer information of the i-th frame image according to the converted depth value.

[0095] In a possible implementation, the i-th frame image includes J pixels, the depth value of the i-th frame image includes the depth values ​​of J pixels, the depth values ​​of J pixels correspond to the coordinate system where the DTOF camera is located, the coordinate system where the DTOF camera is located is the second coordinate system, and the coordinate system where the RGB value of the i-th frame image is located is the first coordinate system; accordingly, the process of the electronic device determining the layer information of the i-th frame image according to the RGB value and depth value of the i-th frame image in the N-frame image can be: converting the depth value corresponding to the j-th pixel in the i-th frame image from the second coordinate system to the first coordinate system to obtain the first RGBD information, and the first RGBD information includes the depth value of each pixel in the i-th frame image in the first coordinate system. Wherein, j takes any positive integer between 1 and J; according to the first RGBD information, the layer information of the i-th frame image is determined. In this way, by aligning the RGB value and the depth value of the i-th frame image, the first RGBD information is obtained, so that the layer information of the i-th frame image is determined more accurately according to the first RGBD information.

[0096] Specifically, the electronic device obtains an intensity map of the first scene through a TOF camera, and obtains an RGB map of the first scene through an RGB camera, calibrates the external parameters of the DTOF camera and the RGB camera according to the intensity map and the RGB map, and obtains the rotation matrix and the translation matrix between the DTOF camera and the RGB camera; according to the rotation matrix and the translation matrix, the depth value corresponding to the jth pixel in the i-th frame image is converted from the second coordinate system where the DTOF camera is located to the first coordinate system where the RGB camera is located, and obtains the above-mentioned first RGBD information. Among them, the first scene can be, for example, a checkerboard scene, and the embodiments of the present application do not make specific restrictions. Among them, the external parameters of the RGB camera refer to the parameters used in the process of converting the RGB value from the world coordinate system to the camera coordinate system, and the external parameters of the DTOF camera refer to the parameters used in the process of converting the depth value from the world coordinate system to the camera coordinate system.

[0097] In a possible implementation, the coordinate system where the RGB value of the i-th frame image is located is the first coordinate system, and the depth value of each pixel in the i-th frame image in the first coordinate system includes M depth levels; when M=2, the layer information indicates that the i-th frame image includes a foreground layer and a background layer; when M>2, the layer information indicates that the i-th frame image includes a foreground layer, a middle ground layer, and a background layer. In this way, for images with complex depth values, the image can be segmented into at least one of the foreground layer, the middle ground layer, or the background layer according to the depth levels in the image, so that the scene segmentation accuracy of the image is higher, thereby improving the image processing effect of complex images.

[0098] Optionally, the depth values ​​of each pixel in the i-th frame image in the first coordinate system can be counted in the form of a histogram. Correspondingly, when the i-th frame image corresponds to the second histogram, one of the M depth levels corresponds to one of the first M depth values ​​with the largest count value in the second histogram. For example, M=2, and the first M depth values ​​with the largest count value in the second histogram of image 1 include depth value 1 and depth value 2. Then, according to the depth levels corresponding to depth value 1 and depth value 2, image 1 is divided into a foreground layer and a background layer. For another example, M=3, and the first M depth values ​​with the largest count value in the second histogram of image 2 include depth value 1, depth value 2, and depth value 3. Then, according to the depth levels corresponding to depth value 1, depth value 2, and depth value 3, image 2 is divided into a foreground layer, an intermediate layer 1, and a background layer. For another example, M=4, and the first M depth values ​​with the largest count values ​​in the second histogram of image 3 include depth value 1, depth value 2, depth value 3, and depth value 4. Then, according to the depth levels corresponding to depth value 1, depth value 2, depth value 3, and depth value 4, image 3 is segmented into a foreground layer, a middle layer 1, a middle layer 2, and a background layer.

[0099] S404: performing image processing on the N frames of images based on the N layer information corresponding to the N frames of images.

[0100] In S404, the electronic device performs image processing on the N frames of images based on the N layer information corresponding to the N frames of images, including but not limited to the following implementations:

[0101] In implementation mode 1, the electronic device determines the region of interest ROI in N frames of images based on N layer information; determines the first feature point in the ROI in the N frames of images; and performs image registration based on the first feature point. In this way, the electronic device can accurately extract the first feature point in the ROI in the N frames of images based on the N layer information, and then can achieve effective image registration based on the first feature point, thereby effectively improving the effect of image registration.

[0102] In implementation mode 2, the electronic device determines the layer where each object in the N frames of images is located based on the N layer information; and performs target recognition on each object in the N frames of images according to the layer where each object is located, and forms a contour frame diagram of each target object in the N frames of images. In this way, the accuracy of target recognition can be improved.

[0103] In implementation mode 3, the electronic device determines the layer where the target object in the N frames of images is located based on the N layer information; and blurs the background of the layers other than the layer where the target object is located, so that the target object can be clearly displayed in the N frames of images. In this way, the effect of blurring the background can be effectively improved.

[0104] Optionally, the electronic device may also send N layer information to the image processing device for the image processing device to perform image processing (e.g., image background blur, target recognition, etc.). Accordingly, the image processing device receives N layer information corresponding to N frames of images; the N layer information is used to determine the region of interest ROI in the N frames of images; the first feature point in the ROI is determined in the N frames of images; and image registration is performed based on the first feature point.

[0105] To facilitate understanding, the technical solutions of the embodiments of the present application are further described in detail below with reference to specific examples.

[0106] See also Figure 5 , Figure 5 An image processing system provided in an embodiment of the present application is shown, which includes a DTOF camera, an RGB camera, a DTOF preprocessing module, an RGB preprocessing module, an ISP module, and a dual-camera fusion processing module.

[0107] Among them, the DTOF camera can obtain the depth value of N frames of images, and input the depth value of N frames of images into the DTOF preprocessing module; the DTOF preprocessing module preprocesses the depth value of N frames of images to obtain the preprocessed depth value; the RGB camera obtains the RGB value of N frames of images, and inputs the RGB value of N frames of images into the DTOF preprocessing module; the DTOF preprocessing module preprocesses the RGB value of N frames of images (such as denoising) to obtain the preprocessed RGB value; the dual-camera fusion processing module fuses the depth value and RGB value of the preprocessed N frames of images to obtain N layer information; the ISP module performs image processing on the N frames of images based on the N layer information.

[0108] See also Figure 6 , when the i-th frame image in N frames includes J pixels, the depth value of the i-th frame image in N frames corresponds to J histograms, and the process of preprocessing the depth value of the N frames image by the DTOF preprocessing module is as follows:

[0109] Step A1: denoising.

[0110] Specifically, the DTOF preprocessing module first performs a Binning operation on the J histograms corresponding to the i-th frame image in the N frames of images to obtain the distribution state of the count values ​​in the J histograms; then, the count values ​​in the J histograms are sorted (for example, from small to large, or from large to small), and the noise in the J histograms is estimated using the median method, and the corresponding noise is removed from the J histograms to obtain the J first histograms after denoising. It should be understood that here the depth values ​​of J pixels are represented by J histograms.

[0111] Among them, the sorting operation usually consumes more computing resources. The DTOF preprocessing module performs Binning operations on J histograms, which can reduce the dimension of the count value, thereby reducing the computational complexity of the sorting operation. In addition, the DTOF preprocessing module performs denoising on the J histograms corresponding to the i-th frame image, which can effectively improve the accuracy of the depth value of the i-th frame image.

[0112] Among them, step A1 is optional.

[0113] Step A2: Depth calculation.

[0114] In a possible implementation, the i-th frame image in the N-frame image includes J pixels, and the depth value of the i-th frame image includes the depth value of J pixels; accordingly, the DTOF preprocessing module receives the first echo data from the DTOF camera, the first echo data includes J first histograms, and the j-th first histogram in the J first histograms is used to characterize the echo position corresponding to the j-th pixel in the i-th frame image; wherein, the j takes any positive integer between 1 and the J; when there are multiple echo positions of the j-th pixel in the j-th first histogram, the frequency of each echo position in the multiple echo positions in the echo position of the first pixel (i.e., multiple pixels adjacent to the j-th pixel in the spatial domain of the first histogram) is counted, and the echo position with the highest frequency among the multiple echo positions is used as the echo position corresponding to the j-th pixel; according to the echo position corresponding to the j-th pixel, the depth value of the j-th pixel is determined. In addition, when there is only one echo position in the j-th first histogram, the echo position is used as the echo position corresponding to the j-th pixel.

[0115] In this way, the accuracy of the echo position of the pixel can be effectively improved, so that the depth value of the pixel is more accurate, and thus the layer information subsequently determined according to the depth value of the pixel is more accurate.

[0116] For example, J=500, that is, the i-th frame image includes 500 pixels, the first echo data includes 500 first histograms, the first first histogram of the 500 first histograms is used to characterize the echo position corresponding to the first pixel in the i-th frame image, the second first histogram of the 500 first histograms is used to characterize the echo position corresponding to the second pixel in the i-th frame image, ..., and so on, the 500th first histogram of the 500 first histograms is used to characterize the echo position corresponding to the 500th pixel in the i-th frame image; wherein, the first histograms corresponding to pixels 1 to 200 each have only one echo position, the first histogram corresponding to pixel 201 has only three echo positions, the first histogram corresponding to pixels 202 to 400 each have only one echo position, the first histogram corresponding to 401 has only five echo positions, and the first histogram corresponding to pixels 402 to 500 each have only one echo position; accordingly, D The TOF preprocessing module uses the echo positions in the first histogram corresponding to pixels 1 to 200, pixels 202 to 400, and pixels 402 to 500 as their own echo positions; in addition, the adjacent pixels of pixel 201 include the pixels within the 3*3 rectangle centered on pixel 201, and the DTOF preprocessing module counts the frequency of each of the three echo positions in the first histogram corresponding to pixel 201 appearing in the echo positions of the pixels within the 3*3 rectangle, and uses the echo position with the highest frequency among the three echo positions as the echo position corresponding to pixel 201; the adjacent pixels of pixel 401 include the pixels within the 5*5 rectangle centered on pixel 401, and the DTOF preprocessing module counts the frequency of each of the five echo positions in the first histogram corresponding to pixel 401 appearing in the echo positions of the pixels within the 5*5 rectangle, and uses the echo position with the highest frequency among the five echo positions as the echo position corresponding to pixel 401.

[0117] Among them, the J first histograms are generated by the DTOF preprocessing module according to the flight time of each laser emitted by the laser sensor in the DTOF camera. Accordingly, the DTOF preprocessing module can determine the depth value of the jth pixel according to the echo position corresponding to the jth pixel. The process includes: the DTOF preprocessing module can determine the depth value of the jth pixel according to the flight time of the laser received by the echo position corresponding to the jth pixel, the time when the DTOF camera detects the echo photon corresponding to the echo position, and the speed of light.

[0118] Step A3: depth correction.

[0119] In a possible implementation, after the DTOF preprocessing module determines the depth value corresponding to the jth pixel according to the echo position corresponding to the jth pixel, the DTOF preprocessing module can also determine the first space normal vector between the jth pixel and the optical center of the DTOF camera according to the depth value corresponding to the jth pixel and the internal parameters of the DTOF camera; when the H angles between the first space normal vector and the second space normal vector are greater than the first threshold, the depth value of the jth pixel is interpolated and corrected according to the depth values ​​of the adjacent pixels of the jth pixel. It should be understood that the adjacent pixels of the jth pixel may include G pixels, and one pixel in the G pixels corresponds to a space normal vector; accordingly, the second space normal vector includes G space normal vectors between the adjacent pixels and the optical center of the DTOF camera. Wherein, H is an integer greater than or equal to 1, and G is an integer greater than or equal to H.

[0120] In this way, the DTOF preprocessing module can interpolate and correct the depth values ​​of some pixels with inaccurate depth values, and can eliminate the error effects of device manufacturing process, high reflectivity scenes, or strong light on the depth value of the i-th frame image, thereby further improving the accuracy of the depth value of the i-th frame image.

[0121] The internal parameters of the DTOF camera are parameters related to the physical characteristics of the DTOF camera itself (such as lens distortion, focal length, etc.). The interpolation method can be bilinear interpolation or biquadratic interpolation, which is not specifically limited in the embodiments of the present application.

[0122] For example, the first threshold is 0.15 degrees, H=G, for the j-th pixel, a 3*3 window is constructed with the j-th pixel as the center, and the pixels in the 3*3 window except the j-th pixel are the adjacent pixels of the j-th pixel, then G=8; the DTOF preprocessing module determines that the first spatial normal vector between the j-th pixel and the optical center of the DTOF camera is spatial normal vector 1 according to the depth value of the j-th pixel and the internal parameters of the DTOF camera; and the DTOF preprocessing module determines that the second spatial normal vector between the adjacent pixel and the optical center of the DTOF camera includes spatial normal vector 2, spatial normal vector 3, spatial normal vector 4, and spatial normal vector 5 according to the depth value of the adjacent pixel of the j-th pixel and the internal parameters of the DTOF camera. Vector 5, spatial normal vector 6, spatial normal vector 7, spatial normal vector 8, spatial normal vector 9; further, the DTOF preprocessing module determines that the angles between spatial normal vector 1 and each of spatial normal vector 2, spatial normal vector 3, spatial normal vector 4, spatial normal vector 5, spatial normal vector 6, spatial normal vector 7, spatial normal vector 8, and spatial normal vector 9 are angle 1, angle 2, angle 3, angle 4, angle 5, angle 6, angle 7, and angle 8 respectively; among which, angle 1, angle 2, angle 3, angle 4, angle 5, angle 6, angle 7, and angle 8 are all greater than 0.15 degrees, the DTOF preprocessing module performs interpolation correction on the depth value of the j-th pixel according to the depth values ​​of all adjacent pixels of the j-th pixel.

[0123] For another example, the first threshold is 0.20 degrees, H=10, for the jth pixel, a 5*5 window is constructed with the jth pixel as the center, and the pixels in the 5*5 window except the jth pixel are the neighboring pixels of the jth pixel, then G=24; the DTOF preprocessing module determines that the first spatial normal vector between the jth pixel and the optical center of the DTOF camera is the spatial normal vector 1 according to the depth value of the jth pixel and the internal parameters of the DTOF camera; and the DTOF preprocessing module determines that the first spatial normal vector between the jth pixel and the optical center of the DTOF camera is the spatial normal vector 1 according to the depth value of the neighboring pixels of the jth pixel and the internal parameters of the DTOF camera. The second spatial normal vector between the adjacent pixel and the optical center of the DTOF camera includes spatial normal vector 2-spatial normal vector 25; the DTOF preprocessing module determines that the angles between spatial normal vector 1 and each item in spatial normal vector 2-spatial normal vector 25 are angle 1, angle 2, angle 3, angle 4, ..., and angle 24 respectively; among which, the 10 angles of angle 1-angle 10 are all greater than 0.15 degrees, then the DTOF preprocessing module interpolates and corrects the depth value of the j-th pixel according to the depth values ​​of all adjacent pixels of the j-th pixel.

[0124] Step A4: Determine the confidence map and the intensity map.

[0125] In some possible embodiments, the DTOF preprocessing module may also receive the echo signal corresponding to the i-th frame image, and obtain the number of echo photons corresponding to each pixel in the i-th frame image according to the echo signal corresponding to the i-th frame image, and generate the confidence of each pixel; and, according to the confidence of each pixel, determine the confidence map of the i-th frame image, and the confidence map is used to reflect the credibility of the depth value of each pixel. And, the DTOF preprocessing module may also obtain the number of echo photons corresponding to all pixels in the i-th frame image according to the echo signal corresponding to the i-th frame image; and generate the intensity map of the i-th frame image according to the number of echo photons corresponding to all pixels, and the intensity map is used to reflect the contour and / or edge of the target object in the i-th frame image.

[0126] In this way, by generating the confidence map and intensity map of the i-th frame image, the image processing effect can be improved when the i-th frame image is subsequently processed according to the confidence map and intensity map. For example, when the DTOF preprocessing module performs target recognition on the i-th frame image based on the confidence map and intensity map, it can only mark the target objects with a confidence greater than 0.9 in the N-frame image, which can effectively improve the accuracy of target recognition.

[0127] See also Figure 7 The dual-camera fusion processing module fuses the depth values ​​of N frames of images and the RGB values ​​of N frames of images to obtain the layer information. The process includes:

[0128] Step B1: calibrate the external parameters of the DTOF camera and the RGB camera.

[0129] Exemplarily, first, a DTOF camera is used to obtain an intensity map of the checkerboard scene, and an RGB camera is used to obtain an RGB map of the checkerboard scene. The dual-camera fusion processing module calibrates the external parameters of the DTOF camera and the RGB camera according to the intensity map and the RGB map to obtain the rotation matrix and translation matrix between the DTOF camera and the RGB camera.

[0130] Step B2: RGBD alignment.

[0131] Specifically, the dual-camera fusion processing module converts the depth value corresponding to the jth pixel in the i-th frame image from the second coordinate system where the DTOF camera is located to the first coordinate system where the RGB camera is located according to the rotation matrix and the translation matrix to obtain the first RGBD information. The i-th frame image includes J pixels, j is a positive integer, 0<j≤J.

[0132] Furthermore, considering that the i-th frame image is affected by the external parameters of the DTOF camera and / or the external parameters of the RGB camera, and the depth values ​​of some pixels in the i-th frame image are missing, after the dual-camera fusion processing module obtains the first RGBD information, the dual-camera fusion processing module can also interpolate the depth value of the second pixel in the i-th frame image according to the depth value of the third pixel in the i-th frame image when the depth value of the second pixel is missing, to obtain the second RGBD information; wherein the third pixel is adjacent to the second pixel, and the confidence corresponding to the depth value of the third pixel is greater than the second threshold. The confidence corresponding to the depth value of each pixel of the i-th frame image can be determined according to the confidence map of the i-th frame image.

[0133] For example, taking the second threshold as 0.9, the depth value of pixel 1 in the i-th frame image is missing, and the adjacent pixels of pixel 1 include pixel 2, pixel 3, and pixel 4; wherein, the confidence levels corresponding to the depth values ​​of pixel 2, pixel 3, and pixel 4 are 0.95, 0.85, and 0.79, respectively, and only the confidence level corresponding to the depth value of pixel 2 is greater than the second threshold, then the electronic device interpolates the depth value of pixel 1 based on the depth value of pixel 2.

[0134] Step B3: Determine the layer information of the i-th frame image according to the second RGBD information.

[0135] Specifically, the dual-camera fusion processing module determines the histogram based on the second RGBD information. Figure 1 ; Among them, the histogram Figure 1 It is used to represent the depth value of each pixel value in the i-th frame image. Figure 1 It includes M depth levels; when M=2, the layer information includes the foreground layer and the background layer of the i-th frame image; when M>2, the layer information includes the foreground layer, the middle layer and the background layer of the i-th frame image. In this way, the scene segmentation accuracy of the image can be higher, thereby improving the image processing effect of complex images.

[0136] In some possible embodiments, the layer information corresponding to two adjacent frames of images in N frames of images is used to determine the motion information between the two adjacent frames of images; wherein the motion information includes global motion information and / or local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the two adjacent frames of images, and the local motion information is used to characterize the motion state of the target object in the two adjacent frames of images. In this way, when the motion information is used for image processing, it can effectively improve the image processing effect (such as image registration, video stabilization, etc.). For example, when the motion information is used for video stabilization, it can effectively describe the real movement of the image acquisition device in three-dimensional space, thereby effectively improving the video stabilization effect in scenes where the image acquisition device moves faster. For another example, when the motion information is used for video stabilization, it can effectively describe the motion state of the target object in the image in three-dimensional space, thereby effectively improving the video stabilization effect in scenes where the depth of the object in the image changes.

[0137] For example, see Figure 8 The dual-camera fusion processing module can determine the layer information of the i+1th frame image according to the RGB value of the i+1th frame image and the depth value of the i+1th frame image; determine the motion information between the i-th frame image and the i+1th frame image according to the layer information of the i-th frame image and the layer information of the i+1th frame image; wherein the motion information includes global motion information and local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the i-th frame image and the i+1th frame image, and the local motion information is used to characterize the motion state of the target object in the i-th frame image and the i+1th frame image. Further, the layer information of the i-th frame image is determined according to histogram A, which is used to characterize the depth values ​​of each pixel value in the i-th frame image, and histogram A includes M depth levels; the layer information of the i+1-th frame image is determined according to histogram B, which is used to characterize the depth values ​​of each pixel value in the i+1-th frame image, and histogram B includes M depth levels; accordingly, the process of the electronic device determining the motion information between the i-th frame image and the i+1-th frame image according to the layer information of the i-th frame image and the layer information of the i+1-th frame image can be: comparing the M depth levels included in the histogram A with the M depth levels included in the histogram B, and if the difference between the two is less than the third threshold, it is considered that the global motion state from the i-th frame image to the i+1-th frame image is a stationary state, otherwise, it is considered that the global motion state is a moving state.

[0138] For example, M=2, and the third threshold is 0.5. The depth values ​​corresponding to the first depth level and the second depth level included in histogram A are 5 and 3 respectively, and the depth values ​​corresponding to the first depth level and the second depth level included in histogram B are 4 and 2 respectively; wherein, the difference between the depth value corresponding to the first depth level included in histogram A and the first depth level included in histogram B is 1, and the difference between the depth value corresponding to the second depth level included in histogram A and the second depth level included in histogram B is 1, and both differences are greater than the third threshold, then it is considered that the global motion state from the i-th frame image to the i+1-th frame image is a motion state. For another example, M=2, and the third threshold is 1. The depth values ​​corresponding to the first depth level and the second depth level included in histogram A are 5 and 3 respectively, and the depth values ​​corresponding to the first depth level and the second depth level included in histogram B are 4.5 and 2.5 respectively; wherein, the difference between the depth value corresponding to the first depth level included in histogram A and the first depth level included in histogram B is 0.5, and the difference between the depth value corresponding to the second depth level included in histogram A and the second depth level included in histogram B is 0.5, and both differences are less than the third threshold, then it is considered that the global motion state from the i-th frame image to the i+1-th frame image is a stationary state.

[0139] Similarly, the electronic device determines the region of interest ROI of the i-th frame image according to the layer information of the i-th frame image; determines the ROI of the i+1-th frame image according to the layer information of the i+1-th frame image; and determines the first N depth levels with higher depth values ​​of the ROI of the i-th frame image in histogram A, determines the first N depth levels with higher depth values ​​of the ROI of the i+1-th frame image in histogram B, compares the first N depth levels included in the histogram A with the first N depth levels included in the histogram B, and if the difference between the two is less than a fourth threshold, it is considered that the local motion state from the i-th frame image to the i+1-th frame image is a stationary state, otherwise, it is considered that the local motion state is a moving state.

[0140] For example, N=3, and the fourth threshold is 0.5 as an example. The first N depth levels with higher depth values ​​of the ROI of the i-th frame image in the histogram A include depth level 1, depth level 2, and depth level 3, and the depth values ​​corresponding to depth level 1, depth level 2, and depth level 3 are 5, 4, and 3, respectively; the first N depth levels with higher depth values ​​of the ROI of the i+1-th frame image in the histogram B include depth level 4, depth level 5, and depth level 6, and the depth values ​​corresponding to depth level 4, depth level 5, and depth level 6 are 3, 2, and 1, respectively; the difference between depth level 1 and depth level 4 is 2, the difference between depth level 2 and depth level 5 is 2, and the difference between depth level 3 and depth level 6 is 2, then these three differences are all greater than the fourth threshold, and it is considered that the local motion information from the i-th frame image to the i+1-th frame image is in a motion state. For another example, N=3, and the fourth threshold is 0.5. The first N depth levels with higher depth values ​​of the ROI of the i-th frame image in the histogram A include depth level 1, depth level 2, and depth level 3, and the depth values ​​corresponding to depth level 1, depth level 2, and depth level 3 are 3.2, 2.2, and 1.2, respectively; the first N depth levels with higher depth values ​​of the ROI of the i+1-th frame image in the histogram B include depth level 4, depth level 5, and depth level 6, and the depth values ​​corresponding to depth level 4, depth level 5, and depth level 6 are 3, 2, and 1, respectively; the difference between depth level 1 and depth level 4 is 0.2, the difference between depth level 2 and depth level 5 is 0.2, and the difference between depth level 3 and depth level 6 is 0.2. These three differences are all less than the fourth threshold, and it is considered that the local motion information from the i-th frame image to the i+1-th frame image is in a stationary state. It should be understood that the value of i is greater than or equal to 1 and less than or equal to N.

[0141] In some possible embodiments, the dual-camera fusion processing module can also determine 3D reprojection information between two adjacent frames of images; wherein the 3D reprojection information is used to screen the target feature point pairs between the two adjacent frames of images. In this way, when the 3D reprojection information is applied to video stabilization, it can effectively describe the motion state of the target object in the image in the three-dimensional space, thereby effectively improving the effect of video stabilization. For example, see Fig. 9, the dual-camera fusion processing module transforms the pixel coordinates of the 2D feature points of the i-th frame image in the N-frame image to obtain the first 3D space coordinates; transforms the pixel coordinates of the 2D feature points of the i+1-th frame image in the N-frame image to obtain the second 3D space coordinates; according to the first 3D space coordinates and the second 3D space coordinates, determines the 3D reprojection information between the i-th frame image and the i+1-th frame image, and the 3D reprojection information is used to screen the target feature point pairs between the i-th frame image and the i+1-th frame image. Among them, the target feature point pair is a set of matching feature points between the i-th frame image and the i+1-th frame image, and the reprojection error between the matching feature points is less than a preset threshold. In this way, the 3D reprojection information between adjacent frames can be determined. When the 3D reprojection information is applied to video stabilization, it can effectively describe the motion state of the target object in the image in the three-dimensional space, thereby effectively improving the effect of video stabilization.

[0142] In some other possible embodiments, the dual-camera fusion processing module can also transform the 2D feature point pixel coordinates of the long frame image in the N frames to obtain a third 3D space coordinate; transform the 2D feature point pixel coordinates of the short frame image in the N frames to obtain a fourth 3D space coordinate; determine the 3D reprojection information between the long frame image and the short frame image according to the third 3D space coordinate and the fourth 3D space coordinate, and the 3D reprojection information is used to screen the target feature point pairs between the long frame image and the short frame image. In this way, the 3D reprojection information between the long frame image and the short frame image can be determined, and when the 3D reprojection information is applied to image registration, it can effectively describe the motion state of the long frame image and the short frame image in three-dimensional space, thereby effectively improving the effect of image registration.

[0143] Optionally, after the electronic device determines the 3D reprojection information, it can also send the 3D reprojection information to the image processing device. Accordingly, the image processing device receives the 3D reprojection information of N frames of images, and the 3D reprojection information is used to screen the target feature point pairs between the i-th frame image and the i+1-th frame image in the N frames of images; based on the target feature point pairs, the relative position and posture between the i-th frame image and the i+1-th frame image is determined, and the relative position and posture is used to determine the motion trajectory of the image acquisition device corresponding to the N frames of images; based on the motion trajectory, video stabilization is performed. In this way, the image processing device can determine the real motion trajectory of the image acquisition device in three-dimensional space according to the received 3D reprojection information. In a possible implementation, the 3D reprojection information includes the reprojection error between the i-th frame image and the i+1-th frame image; accordingly, the electronic device determines the relative position and posture between the i-th frame image and the i+1-th frame image according to the 3D reprojection information, including: when the reprojection error is less than the fifth threshold, the relative position and posture between the i-th frame image and the i+1-th frame image is determined according to the first 3D space coordinate and the second 3D space coordinate.

[0144] In the embodiment of the present application, the ISP module performs image processing on N frame images based on N layer information. There are multiple implementation methods, which are introduced below using video stabilization and image registration as examples.

[0145] Example 1: Video stabilization.

[0146] See also Fig.10 , Fig.10 FIG. 2 shows a schematic diagram of the video anti-shake process. Fig.10 In the process (a), the original feature points of N frames of images contain the motion information (i.e., local motion information) of the target object (e.g., a person or car moving in the picture) in the N frames of images. The ISP module can accurately identify the motion area of ​​the target object in the N frames of images based on the layer information determined by the dual-camera fusion processing module, process the feature points in the area, obtain feature points that conform to the original motion of the image acquisition device (e.g., camera), calculate the motion trajectory of the image acquisition device in a two-dimensional plane, and display N frames of images based on the motion trajectory, thereby improving the video stabilization effect of complex scenes. Alternatively, the ISP module determines the two-dimensional feature points that characterize the motion of the moving image acquisition device (e.g., camera) in the N frames of images based on the global motion information determined by the dual-camera fusion processing module, calculates the motion trajectory of the image acquisition device in a two-dimensional plane, and displays N frames of images based on the motion trajectory, thereby improving the video stabilization effect of complex scenes. In this way, the problem of abnormal video stabilization can be solved. Among them, complex scenes include at least one of the following: scenes where the movement direction and / or speed of the image acquisition device and the target object in the image are inconsistent, scenes where the image has many depth levels, scenes where the image acquisition device moves faster, or scenes where the depth of the target object in the image changes greatly.

[0147] exist Fig.10 In process (b), the ISP module screens the target feature point pairs between the i-th frame image and the i+1-th frame image in N frames of images based on the 3D reprojection information (i.e., screens in the two-dimensional feature points); determines the relative pose between the i-th frame image and the i+1-th frame image based on the target feature point pairs; and determines the motion trajectory of the image acquisition device corresponding to the N frames of images based on the relative pose; and performs video stabilization based on the motion trajectory. Optionally, the 3D reprojection information includes the reprojection error between the i-th frame image and the i+1-th frame image; accordingly, when the reprojection error is less than a fifth threshold, the ISP module determines the relative pose between the i-th frame image and the i+1-th frame image based on the first 3D space coordinates and the second 3D space coordinates. In this way, the image processing device can determine the actual motion trajectory of the image acquisition device in the three-dimensional space based on the received 3D reprojection information, thereby improving the overall video stabilization effect.

[0148] Example 2: Image registration.

[0149] See also Fig.11 , the ISP module can determine the region of interest ROI in image frame 1 based on the layer information corresponding to image frame 1; determine feature point 1 in ROI in image frame 1; and the ISP module can determine the region of interest ROI in image frame 2 based on the layer information corresponding to image frame 2; determine feature point 2 in ROI in image frame 2; the ISP module performs feature point matching based on feature point 1 and feature point 2; and performs motion estimation based on the matched feature points to obtain optical flow; and performs image registration based on the optical flow. Among them, the region of interest ROI can be, for example, the area where the target object is located.

[0150] The ISP module performs image processing on N frames of images, and there are other implementation methods. For example, using layer information for image background blur can improve the image background blur effect; for another example, using 3D reprojection information for three-dimensional reconstruction, and according to the relative posture determined by the 3D reprojection information, trajectory constraints between adjacent image frames can be established, thereby improving the accuracy of three-dimensional reconstruction. For another example, according to the layer information, feature points of N frames of images are screened, and the corresponding areas of the long exposure frame and the short exposure frame can be accurately fused, thereby improving the High Dynamic Range Imaging (HDR) effect.

[0151] According to the above method, Fig.12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the device may be an electronic device, or a chip or circuit, such as a chip or circuit that can be set in an electronic device. The electronic device may be a device with processing capabilities, such as Figure 1 The electronic device 130.

[0152] Furthermore, the image processing device 1201 may further include a bus system, wherein the processor 1202 , the memory 1204 , and the transceiver 1203 may be connected via the bus system.

[0153] It should be understood that the processor 1202 may be a chip. For example, the processor 1202 may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0154] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 1202 or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor 1202 for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1204, and the processor 1202 reads the information in the memory 1204 and completes the steps of the above method in conjunction with its hardware.

[0155] It should be noted that the processor 1202 in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0156] It can be understood that the memory 1204 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0157] When the image processing device 1201 corresponds to an electronic device, the image processing device may include a processor 1202, a transceiver 1203, and a memory 1204. The memory 1204 is used to store instructions, and the processor 1202 is used to execute the instructions stored in the memory 1204 to implement the relevant scheme of the electronic device in the above method embodiment.

[0158] Based on the above embodiments and the same concept, Fig.13 A schematic diagram of an image processing device provided in an embodiment of the present application, such as Fig.13 As shown, the image processing device 1300 may be an electronic device, or a chip or circuit in an electronic device. The image processing device 1300 may implement the steps performed by the electronic device as described above.

[0159] The image processing device 1300 may include an acquisition unit 1301, which is used to acquire RGB values ​​of N frames of images; and, to acquire depth values ​​of the N frames of images; wherein N is an integer greater than or equal to 1; a processing unit 1302, which is used to determine layer information of the i-th frame of image according to the RGB value of the i-th frame of image and the depth value of the i-th frame of image in the N frames of image; wherein i is any positive integer between 1 and N, and the layer information indicates the layer included in the i-th frame of image; the processing unit is also used to perform image processing on the N frames of image based on the N layer information corresponding to the N frames of image.

[0160] In the embodiment of the present application, the layer included in the i-th frame image may be at least one of a foreground layer, a background layer, or a mid-ground layer. The mid-ground layer includes one or more intermediate layers other than the foreground layer and the background layer obtained when the i-th frame image is subjected to scene segmentation.

[0161] In one possible design, the device also includes a receiving unit 1303; the receiving unit 1303 is used to receive the echo signal corresponding to the i-th frame image; the processing unit 1302 is also used to obtain the number of echo photons corresponding to each pixel in the i-th frame image according to the echo signal corresponding to the i-th frame image, and generate the confidence of each pixel; and, according to the confidence of each pixel, determine the confidence map of the i-th frame image, the confidence map is used to reflect the credibility of the depth value of each pixel.

[0162] In one possible design, the image processing device also includes a receiving unit 1303; the receiving unit 1303 is used to receive the echo signal corresponding to the i-th frame image; the processing unit 1302 is also used to obtain the number of echo photons corresponding to all pixels in the i-th frame image according to the echo signal corresponding to the i-th frame image; based on the number of echo photons corresponding to all pixels, an intensity map of the i-th frame image is generated, and the intensity map is used to reflect the contour and / or edge of the target object in the i-th frame image.

[0163] In a possible design, layer information corresponding to two adjacent frames of images in N frames of images is used to determine motion information between the two adjacent frames of images; wherein the motion information includes global motion information and / or local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the two adjacent frames of images, and the local motion information is used to characterize the motion state of the target object in the two adjacent frames of images.

[0164] In a possible design, the processing unit 1302 is further used to: determine 3D reprojection information between two adjacent frames of images; wherein the 3D reprojection information is used to screen target feature point pairs between two adjacent frames of images.

[0165] In one possible design, the apparatus further includes a sending unit 1304, and the sending unit 1304 is configured to send 3D re-projection information to an image processing device.

[0166] In a possible design, N layers of information are used to determine a region of interest ROI in N frames of images, and the first feature point in the ROI is used for image registration.

[0167] In one possible design, the sending unit 1304 is also used to send N layers of information to the image processing device.

[0168] In a possible design, the coordinate system where the RGB values ​​of the i-th frame image are located is the first coordinate system, and the depth values ​​of each pixel in the i-th frame image in the first coordinate system include M depth levels; when M=2, the layer information indicates that the layers of the i-th frame image include foreground layers and background layers; when M>2, the layer information indicates that the layers of the i-th frame image include foreground layers, mid-ground layers, and background layers. Among them, the mid-ground layer may include one or more intermediate layers, and an intermediate layer may be, for example, a layer where an object in the image is located. Optionally, a depth level in the M depth levels corresponds to a depth value in the first M depth values ​​with the largest count value in the second histogram.

[0169] In a possible design, the processing unit 1302 is also used to: when H angles between the first spatial normal vector and the second spatial normal vector are greater than a first threshold, interpolate and correct the depth value of the second pixel according to the depth values ​​of the adjacent pixels of the second pixel; wherein the first spatial normal vector is the first spatial normal vector between the second pixel in the i-th frame image and the image acquisition device of the i-th frame image, the second spatial normal vector includes G spatial normal vectors between the adjacent pixels of the second pixel and the image acquisition device, H is an integer greater than or equal to 1, and G is an integer greater than or equal to H.

[0170] In a possible design, the acquisition unit 1301 is a DTOF camera, that is, the depth values ​​of N frames of images can be acquired through the DTOF camera.

[0171] In a possible design, the i-th frame image includes J pixels, and the depth value of the i-th frame image includes the depth values ​​of J pixels; accordingly, the depth value of the N-frame image is obtained by the DTOF camera, including: receiving the first echo data from the DTOF camera, the first echo data includes J first histograms, and the j-th first histogram in the J first histograms is used to characterize the echo position corresponding to the j-th pixel in the i-th frame image; wherein j is any positive integer between 1 and J; when there are multiple echo positions of the j-th pixel in the j-th first histogram, the frequency of each echo position in the echo position of the first pixel is counted, and the echo position with the highest frequency in the multiple echo positions is used as the echo position corresponding to the j-th pixel; wherein the first pixel includes multiple pixels adjacent to the j-th pixel in the spatial domain of the j-th first histogram; and according to the echo position corresponding to the j-th pixel, the depth value of the j-th pixel is determined. wherein the J first histograms are generated according to the flight time of each laser emitted by the laser sensor in the DTOF camera.

[0172] In a possible design, when the i-th frame image includes J pixels, the depth value of the i-th frame image includes the depth values ​​of J pixels, the depth values ​​of J pixels correspond to the coordinate system where the DTOF camera is located, the coordinate system where the DTOF camera is located is the second coordinate system, and the coordinate system where the RGB value of the i-th frame image is located is the first coordinate system; accordingly, the electronic device determines the layer information of the i-th frame image according to the RGB value and depth value of the i-th frame image in the N-frame image. The process can be: converting the depth value corresponding to the j-th pixel in the i-th frame image from the second coordinate system to the first coordinate system to obtain the first RGBD information, and the first RGBD information includes the depth value of each pixel in the i-th frame image in the first coordinate system. Wherein, j takes any positive integer between 1 and J; according to the first RGBD information, the layer information of the i-th frame image is determined.

[0173] Optionally, the process of converting the depth value of the jth pixel in the i-th frame image from the second coordinate system to the first coordinate system by the electronic device can be: obtaining an intensity map of the first scene through a TOF camera, and obtaining an RGB map of the first scene through an RGB camera, calibrating the external parameters of the DTOF camera and the RGB camera according to the intensity map and the RGB map, and obtaining the rotation matrix and translation matrix between the DTOF camera and the RGB camera; according to the rotation matrix and the translation matrix, converting the depth value corresponding to the jth pixel in the i-th frame image from the second coordinate system where the DTOF camera is located to the first coordinate system where the RGB camera is located, and obtaining the first RGBD information. Among them, the first scene can be, for example, a checkerboard scene, and the embodiments of the present application are not specifically limited. Among them, the external parameters of the RGB camera refer to the parameters used in the process of converting the RGB value from the world coordinate system to the camera coordinate system, and the external parameters of the DTOF camera refer to the parameters used in the process of converting the depth value from the world coordinate system to the camera coordinate system.

[0174] Optionally, after obtaining the first RGBD information, the processing unit 1302 may also interpolate the depth value of the second pixel according to the depth value of the third pixel in the i-th frame image to obtain second RGBD information when the depth value of the second pixel in the i-th frame image is missing; wherein the third pixel is adjacent to the second pixel and the confidence corresponding to the depth value of the third pixel is greater than the second threshold; and determine the layer information of the i-th frame image according to the second RGBD information.

[0175] Based on the above embodiments and the same concept, Fig.14 A schematic diagram of an image processing device provided in an embodiment of the present application, such as Fig.14 As shown, the image processing device 1400 may be an image processing device, or a chip or circuit in the image processing device. The image processing device 1400 may include a communication unit 1401, which is used to receive N scene perception information corresponding to N frames of images; the scene perception information is used to determine the region of interest ROI in the N frames of images; a processing unit 1402, which is used to determine the first feature point corresponding to the ROI in the N frames of images; and the processing unit is also used to perform image registration based on the first feature point.

[0176] In one possible implementation, the communication unit 1401 is further used to receive 3D reprojection information of N frames of images, and the 3D reprojection information is used to screen target feature point pairs between the i-th frame image and the i+1-th frame image in the N frames of images; the processing unit 1402 is further used to determine the relative posture between the i-th frame image and the i+1-th frame image based on the target feature point pairs, and the relative posture is used to determine the motion trajectory of the image acquisition device corresponding to the N frames of images; and, based on the motion trajectory, perform video stabilization.

[0177] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes the method of any one of the above method embodiments.

[0178] An embodiment of the present application also provides an electronic device, which is used to execute a method in any one of the above method embodiments.

[0179] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0180] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0183] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0185] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0186] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing device, characterized in that: include: An acquisition unit, used for acquiring RGB values ​​of N frames of images; And, obtaining the depth value of the N frames of image; wherein N is an integer greater than or equal to 1; A processing unit, configured to determine the layer information of the i-th frame image according to the RGB value of the i-th frame image in the N frames of images and the depth value of the i-th frame image; wherein i is any positive integer between 1 and N, and the layer information indicates the layer included in the i-th frame image; The processing unit is further used to perform image processing on the N frames of images based on the N layer information corresponding to the N frames of images.

2. The device according to claim 1, characterized in that The device also includes a receiving unit; The receiving unit is used to receive the echo signal corresponding to the i-th frame image; The processing unit is further used to obtain the number of echo photons corresponding to each pixel in the i-th frame of image according to the echo signal corresponding to the i-th frame of image, and generate the confidence of each pixel; And, according to the confidence of each pixel, a confidence map of the i-th frame image is determined, and the confidence map is used to reflect the credibility of the depth value of each pixel.

3. The device according to claim 1 or 2, characterized in that The device also includes a receiving unit; The receiving unit is used to receive the echo signal corresponding to the i-th frame image; The processing unit is also used to obtain the number of echo photons corresponding to all pixels in the i-th frame image based on the echo signal corresponding to the i-th frame image; and generate an intensity map of the i-th frame image based on the number of echo photons corresponding to all pixels, wherein the intensity map is used to reflect the contour and / or edge of the target object in the i-th frame image.

4. The device according to any one of claims 1 to 3, characterized in that: The layer information corresponding to two adjacent frames of images in the N frames of images is used to determine the motion information between the two adjacent frames of images; wherein the motion information includes global motion information and / or local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the two adjacent frames of images, and the local motion information is used to characterize the motion state of the target object in the two adjacent frames of images.

5. The device according to any one of claims 1 to 4, characterized in that: The processing unit is also used for: Determine 3D reprojection information between two adjacent frames of images; wherein the 3D reprojection information is used to screen target feature point pairs between the two adjacent frames of images.

6. The device according to any one of claims 1 to 5, characterized in that: The N pieces of layer information are used to determine a region of interest ROI in the N frames of image, and the first feature point in the ROI is used for image registration.

7. The device according to any one of claims 1 to 6, characterized in that: The coordinate system where the RGB values ​​of the i-th frame image are located is a first coordinate system, and the depth value of each pixel in the i-th frame image in the first coordinate system includes M depth levels; When M=2, the layer information indicates that the layers of the i-th frame image include a foreground layer and a background layer; when M>2, the layer information indicates that the layers of the i-th frame image include a foreground layer, a midground layer and a background layer.

8. The device according to any one of claims 1 to 7, characterized in that: The processing unit is also used for: When H angles between the first space normal vector and the second space normal vector are greater than a first threshold, interpolating and correcting the depth value of the second pixel according to depth values ​​of adjacent pixels of the second pixel; Among them, the first spatial normal vector is the first spatial normal vector between the second pixel in the i-th frame image and the image acquisition device of the i-th frame image, the second spatial normal vector includes G spatial normal vectors between adjacent pixels of the second pixel and the image acquisition device, the H is an integer greater than or equal to 1, and the G is an integer greater than or equal to the H.

9. An image processing method, characterized in that: Used in electronic equipment, including: Obtain RGB values ​​of N frames of images; wherein N is an integer greater than or equal to 1; Obtaining the depth values ​​of the N frames of images; Determine the layer included in the i-th frame image according to the RGB value of the i-th frame image in the N-frame image and the depth value of the i-th frame image; wherein i is any positive integer between 1 and N; and the layer information indicates the layer included in the i-th frame image; Based on the N pieces of layer information corresponding to the N frames of images, image processing is performed on the N frames of images.

10. The method according to claim 9, characterized in that The method further comprises: Receiving an echo signal corresponding to the i-th frame image; acquiring the number of echo photons corresponding to each pixel in the i-th frame image according to the echo signal corresponding to the i-th frame image, and generating the confidence of each pixel; A confidence map of the i-th frame image is determined according to the confidence of each pixel, and the confidence map is used to reflect the credibility of the depth value of each pixel.

11. The method according to claim 9 or 10, characterized in that The method further comprises: Receiving an echo signal corresponding to the i-th frame image; acquiring the number of echo photons corresponding to all pixels in the i-th frame image according to the echo signal corresponding to the i-th frame image; An intensity map of the i-th frame image is generated according to the number of echo photons corresponding to all the pixels, and the intensity map is used to reflect the contour and / or edge of the target object in the i-th frame image.

12. The method according to any one of claims 9 to 11, characterized in that: The layer information corresponding to two adjacent frames of images in the N frames of images is used to determine the motion information between the two adjacent frames of images; wherein the motion information includes global motion information and / or local motion information, the global motion information is used to characterize the motion state of the image acquisition device corresponding to the two adjacent frames of images, and the local motion information is used to characterize the motion state of the target object in the two adjacent frames of images.

13. The method according to any one of claims 9 to 12, characterized in that: The method further comprises: Determine 3D reprojection information between two adjacent frames of images; wherein the 3D reprojection information is used to screen target feature point pairs between the two adjacent frames of images.

14. The method according to any one of claims 9 to 13, characterized in that: The N pieces of layer information are used to determine a region of interest ROI in the N frames of image, and the first feature point in the ROI is used for image registration.

15. The method according to any one of claims 9 to 14, characterized in that: The coordinate system where the RGB values ​​of the i-th frame image are located is a first coordinate system, and the depth value of each pixel in the i-th frame image in the first coordinate system includes M depth levels; When M=2, the layer information indicates that the layers of the i-th frame image include a foreground layer and a background layer; when M>2, the layer information indicates that the layers of the i-th frame image include a foreground layer, a midground layer and a background layer.

16. The method according to any one of claims 9 to 15, characterized in that: The method further comprises: When H angles between the first space normal vector and the second space normal vector are greater than a first threshold, interpolating and correcting the depth value of the second pixel according to depth values ​​of adjacent pixels of the second pixel; Among them, the first spatial normal vector is the first spatial normal vector between the second pixel in the i-th frame image and the image acquisition device of the i-th frame image, the second spatial normal vector includes G spatial normal vectors between adjacent pixels of the second pixel and the image acquisition device, the H is an integer greater than or equal to 1, and the G is an integer greater than or equal to the H.

17. An image processing device, characterized in that: The image processing device comprises a processor and a memory, wherein the memory stores program code, and the processor is used to call the program code to execute the method according to any one of claims 9 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed by a computing device, the method as claimed in any one of claims 9 to 16 is implemented.

Images