Image processing method, device, electronic device and storage medium

Abstract

The present application discloses an image processing method, device, electronic device and storage medium, which belongs to the field of computer technology. The method comprises: obtaining multiple image frames captured by a target camera; downsampling the multiple image frames respectively to obtain corresponding multiple image sets; for any image frame, based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame, obtaining the optical flow detection information and depth detection information of the image frame. The present application makes the details contained in the optical flow detection information and depth detection information richer by using the hierarchical image pyramid features in the image set, thereby improving the accuracy of the optical flow detection information and depth detection information, that is, improving the accuracy of the scene flow analysis.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer and mobile communication technologies, autonomous vehicles are gaining widespread attention as an emerging technology. Autonomous vehicles, also known as unmanned vehicles or wheeled mobile robots, are intelligent vehicles controlled by an autonomous driving system.

[0003] In autonomous driving systems, cameras are an important sensor that can capture video streams of the vehicle's surroundings. By performing scene flow analysis (including optical flow analysis and depth prediction) on the video streams, they can be used to perceive the movement of objects outside the vehicle. Therefore, there is an urgent need for a method that can improve the accuracy of scene flow analysis. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium that can improve the accuracy of scene flow analysis. The technical solution is as follows:

[0005] In one aspect, an image processing method is provided, the method comprising:

[0006] Acquire multiple image frames captured by a target camera, wherein the multiple image frames have a temporal correlation relationship;

[0007] Downsampling the multiple image frames respectively to obtain corresponding multiple image sets, each image set including the corresponding image frame and an image obtained by downsampling the image frame;

[0008] For any image frame among the multiple image frames, based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame, optical flow detection information and depth detection information of the image frame are obtained, the optical flow detection information is used to characterize the movement of the same object between the image frame and the previous frame, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera.

[0009] In one possible embodiment, each image set of the multiple image sets includes a corresponding image frame and multiple downsampled images obtained by downsampling the image frame at different sampling multiples, wherein the size of the downsampled image is negatively correlated with the sampling multiple, and the size of the downsampled image is smaller than the size of the image frame.

[0010] In one possible implementation, acquiring the optical flow detection information and the depth detection information of the image frame based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame includes:

[0011] Determining, from an image set corresponding to the image frame and an image set corresponding to a previous frame of the image frame, a plurality of downsampled image pairs corresponding to respective sampling magnifications;

[0012] Based on each of the downsampled image pairs, obtaining intermediate optical flow detection information and intermediate depth detection information;

[0013] Based on the intermediate optical flow detection information and the intermediate depth detection information, optical flow detection information and depth detection information of the image frame are obtained.

[0014] In one possible implementation, obtaining intermediate optical flow detection information and intermediate depth detection information based on each downsampled image pair includes:

[0015] For any sampling magnification, obtaining a first down-sampled image corresponding to the sampling magnification from an image set corresponding to the image frame;

[0016] Acquire a second down-sampled image corresponding to the sampling magnification from an image set corresponding to a previous frame of the image frame;

[0017] Obtaining a correlation matrix based on the first down-sampled image and the second down-sampled image, wherein the correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous frame of the image frame at the sampling magnification;

[0018] Based on the correlation matrix, the intermediate optical flow detection information and the intermediate depth detection information are obtained.

[0019] In one possible implementation, the target camera is a binocular camera, the image frame includes a left-eye image and a right-eye image, the first downsampled image includes a first left-eye downsampled image and a first right-eye downsampled image, and the second downsampled image includes a second left-eye downsampled image and a second right-eye downsampled image;

[0020] The acquiring a correlation matrix based on the first down-sampled image and the second down-sampled image includes:

[0021] Obtaining a first correlation matrix based on the first left-eye downsampled image and the second left-eye downsampled image, wherein the first correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous left-eye image at the sampling magnification;

[0022] Obtaining a second correlation matrix based on the first left-eye downsampled image and the first right-eye downsampled image, wherein the second correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of the image frame at the sampling magnification;

[0023] Obtaining a third correlation matrix based on the second left-eye downsampled image and the second right-eye downsampled image, wherein the third correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of a previous frame of the image frame at the sampling magnification;

[0024] A fourth correlation matrix is acquired based on the second correlation matrix and the third correlation matrix, where the fourth correlation matrix is used to represent a degree of parallax correlation between the image frame and a previous frame of the image frame at the sampling magnification.

[0025] In one possible implementation, obtaining a correlation matrix based on the first down-sampled image and the second down-sampled image includes:

[0026] For each pixel in the first down-sampled image, determining a sliding window that includes the pixel and a plurality of neighboring pixels of the pixel;

[0027] Based on the target step size, sliding sampling is performed on the second down-sampled image according to the sliding window to obtain a correlation coefficient between the pixel and each pixel in the second down-sampled image.

[0028] In one possible implementation, performing sliding sampling on the second down-sampled image according to the sliding window to obtain a correlation coefficient between the pixel and each pixel in the second down-sampled image includes:

[0029] Determine, from the first down-sampled image, a first vector consisting of the pixel and a plurality of neighboring pixels of the pixel;

[0030] For any sliding sampling, determining, from the second down-sampled image, a second vector consisting of a target pixel contained in the sliding window and a plurality of neighboring pixels of the target pixel;

[0031] A dot product between the first vector and the second vector is determined as a correlation coefficient between the pixel and the target pixel.

[0032] In a possible implementation, acquiring the optical flow detection information and the depth detection information of the image frame based on the intermediate optical flow detection information and the intermediate depth detection information includes:

[0033] Acquire a third down-sampled image based on the intermediate optical flow detection information, the intermediate depth detection information, and the second down-sampled image, where the third down-sampled image is a down-sampled image corresponding to the sampling magnification obtained by prediction based on a previous frame of the image frame;

[0034] Upsampling the third downsampled image to obtain an upsampled image corresponding to a target sampling ratio, wherein the target sampling ratio is smaller than and closest to the sampling ratio among the multiple sampling ratios;

[0035] Iteratively performing the step of obtaining intermediate optical flow detection information and intermediate depth detection information on the upsampled image and the downsampled image corresponding to the target sampling ratio in the image set corresponding to the image frame;

[0036] During any iteration, when the size of the obtained up-sampled image is the same as the size of the image frame, the optical flow detection information and the depth detection information obtained based on the up-sampled image and the image frame are output.

[0037] In one possible implementation, downsampling the multiple image frames respectively to obtain corresponding multiple image sets includes:

[0038] For each of the multiple image frames, input the image frame into a plurality of convolutional layers connected in series, and perform convolution processing on the image frame through the plurality of convolutional layers to obtain a plurality of downsampled images, wherein each downsampled image is a feature map output by a convolutional layer;

[0039] The image frame and the multiple down-sampled images are determined as an image set corresponding to the image frame.

[0040] In a possible implementation, the target camera is a vehicle-mounted camera, and the multiple image frames are multiple consecutive video frames in a video stream recorded by the vehicle-mounted camera.

[0041] In one aspect, an image processing apparatus is provided, the apparatus comprising:

[0042] A first acquisition module is used to acquire multiple image frames captured by a target camera, wherein the multiple image frames have a temporal correlation relationship;

[0043] a downsampling module, configured to downsample the plurality of image frames respectively to obtain a plurality of corresponding image sets, each image set including a corresponding image frame and an image obtained by downsampling the image frame;

[0044] The second acquisition module is used to obtain optical flow detection information and depth detection information of any image frame among the multiple image frames based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame, wherein the optical flow detection information is used to characterize the movement of the same object between the image frame and the previous frame, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera.

[0045] In one possible embodiment, each image set of the multiple image sets includes a corresponding image frame and multiple downsampled images obtained by downsampling the image frame at different sampling multiples, wherein the size of the downsampled image is negatively correlated with the sampling multiple, and the size of the downsampled image is smaller than the size of the image frame.

[0046] In one possible implementation, the second acquisition module includes:

[0047] a determination submodule, configured to determine, from an image set corresponding to the image frame and an image set corresponding to a previous frame of the image frame, a plurality of downsampled image pairs corresponding to respective sampling magnifications;

[0048] A first acquisition submodule, configured to acquire intermediate optical flow detection information and intermediate depth detection information based on each downsampled image pair;

[0049] The second acquisition submodule is configured to acquire the optical flow detection information and the depth detection information of the image frame based on the intermediate optical flow detection information and the intermediate depth detection information.

[0050] In one possible implementation, the first acquisition submodule includes:

[0051] A first acquiring unit is configured to acquire, for any sampling magnification, a first down-sampled image corresponding to the sampling magnification from an image set corresponding to the image frame;

[0052] The first acquisition unit is further configured to acquire a second down-sampled image corresponding to the sampling magnification from an image set corresponding to a previous frame of the image frame;

[0053] a second acquiring unit, configured to acquire a correlation matrix based on the first down-sampled image and the second down-sampled image, wherein the correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous frame of the image frame at the sampling magnification;

[0054] A third acquisition unit is configured to acquire the intermediate optical flow detection information and the intermediate depth detection information based on the correlation matrix.

[0055] In one possible implementation, the target camera is a binocular camera, the image frame includes a left-eye image and a right-eye image, the first downsampled image includes a first left-eye downsampled image and a first right-eye downsampled image, and the second downsampled image includes a second left-eye downsampled image and a second right-eye downsampled image;

[0056] The second acquiring unit is configured to:

[0057] Obtaining a first correlation matrix based on the first left-eye downsampled image and the second left-eye downsampled image, wherein the first correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous left-eye image at the sampling magnification;

[0058] Obtaining a second correlation matrix based on the first left-eye downsampled image and the first right-eye downsampled image, wherein the second correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of the image frame at the sampling magnification;

[0059] Obtaining a third correlation matrix based on the second left-eye downsampled image and the second right-eye downsampled image, wherein the third correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of a previous frame of the image frame at the sampling magnification;

[0060] A fourth correlation matrix is acquired based on the second correlation matrix and the third correlation matrix, where the fourth correlation matrix is used to represent a degree of parallax correlation between the image frame and a previous frame of the image frame at the sampling magnification.

[0061] In one possible implementation, the second acquiring unit includes:

[0062] a determining subunit, configured to determine, for each pixel in the first down-sampled image, a sliding window including the pixel and a plurality of neighboring pixels of the pixel;

[0063] A sampling subunit is configured to perform sliding sampling on the second downsampled image according to the sliding window based on a target step size, so as to obtain a correlation coefficient between the pixel and each pixel in the second downsampled image.

[0064] In one possible implementation, the sampling subunit is configured to:

[0065] Determine, from the first down-sampled image, a first vector consisting of the pixel and a plurality of neighboring pixels of the pixel;

[0066] For any sliding sampling, determining, from the second down-sampled image, a second vector consisting of a target pixel contained in the sliding window and a plurality of neighboring pixels of the target pixel;

[0067] A dot product between the first vector and the second vector is determined as a correlation coefficient between the pixel and the target pixel.

[0068] In one possible implementation, the second acquisition submodule is configured to:

[0069] Acquire a third down-sampled image based on the intermediate optical flow detection information, the intermediate depth detection information, and the second down-sampled image, where the third down-sampled image is a down-sampled image corresponding to the sampling magnification obtained by prediction based on a previous frame of the image frame;

[0070] Upsampling the third downsampled image to obtain an upsampled image corresponding to a target sampling ratio, wherein the target sampling ratio is smaller than and closest to the sampling ratio among the multiple sampling ratios;

[0071] Iteratively performing the step of obtaining intermediate optical flow detection information and intermediate depth detection information on the upsampled image and the downsampled image corresponding to the target sampling ratio in the image set corresponding to the image frame;

[0072] During any iteration, when the size of the obtained up-sampled image is the same as the size of the image frame, the optical flow detection information and the depth detection information obtained based on the up-sampled image and the image frame are output.

[0073] In one possible implementation, the downsampling module is configured to:

[0074] For each of the multiple image frames, input the image frame into a plurality of convolutional layers connected in series, and perform convolution processing on the image frame through the plurality of convolutional layers to obtain a plurality of downsampled images, wherein each downsampled image is a feature map output by a convolutional layer;

[0075] The image frame and the multiple down-sampled images are determined as an image set corresponding to the image frame.

[0076] In a possible implementation, the target camera is a vehicle-mounted camera, and the multiple image frames are multiple consecutive video frames in a video stream recorded by the vehicle-mounted camera.

[0077] On the one hand, an electronic device is provided, which includes one or more processors and one or more memories, wherein the one or more memories store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors to implement an image processing method as described in any possible implementation method.

[0078] On the one hand, a storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the image processing method as described in any possible implementation manner.

[0079] In one aspect, a computer program product or computer program is provided, comprising one or more program codes stored in a computer-readable storage medium. One or more processors of an electronic device are capable of reading the one or more program codes from the computer-readable storage medium, and executing the one or more program codes by the one or more processors, thereby enabling the electronic device to perform the image processing method of any of the possible embodiments described above.

[0080] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0081] By downsampling the image frames captured by the target camera, an image set consisting of images in a series of scale spaces is obtained. The images in different scale spaces in the image set are used to comprehensively predict the optical flow detection information and depth detection information of the current image frame. The hierarchical pyramid features in the image set can be used to make the details contained in the optical flow detection information and depth detection information richer, thereby improving the accuracy of the optical flow detection information and depth detection information, that is, improving the accuracy of scene flow analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0083] Figure 1 This is a schematic diagram of an implementation environment of an image processing method provided in an embodiment of the present application;

[0084] Figure 2 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0085] Figure 3 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0086] Figure 4 This is a schematic diagram of the principle of a convolution operation provided in an embodiment of the present application;

[0087] Figure 5 This is a schematic diagram of the principle of a zero-padding method provided in an embodiment of the present application;

[0088] Figure 6 This is a schematic diagram of the principle of a zero-padding method provided in an embodiment of the present application;

[0089] Figure 7 It is a standard optical flow parameter map provided in the embodiment of the present application;

[0090] Figure 8 This is a schematic diagram of the architecture of a scene flow analysis model provided in an embodiment of the present application;

[0091] Figure 9 This is a principle flow chart of an image processing method provided in an embodiment of the present application;

[0092] Figure 10 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0093] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0094] Figure 12 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0096] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0097] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of first positions means two or more first positions.

[0098] The following explains the terms used in the embodiments of the present application:

[0099] Unmanned vehicles (UCVs), also known as self-driving cars or wheeled mobile robots, primarily rely on an in-vehicle intelligent driverless system (AI) based on a computer system to achieve autonomous operation. UCVs are intelligent vehicles that use onboard sensor systems to perceive the road environment, automatically plan routes, and control the vehicle to its intended destination. UCVs use onboard sensors to perceive their surroundings and, based on this information about the road, vehicle position, and obstacles, control steering and speed, enabling safe and reliable road travel. UCVs integrate numerous cutting-edge technologies, including automatic control, architecture, artificial intelligence, and visual computing, and are the product of advanced computer science, pattern recognition, and intelligent control technologies.

[0100] Autonomous Driving System: Utilizing advanced communications, computing, networking, and control technologies, the autonomous driving system provides real-time, continuous control of vehicles. Modern communication methods, directly connected to the vehicle, enable two-way data communication between the vehicle and the ground, with fast transmission rates and large amounts of information. This allows tracking vehicles and the control center to instantly determine the exact location of the preceding vehicle, making operational management more flexible and control more efficient, better suited to the needs of autonomous driving.

[0101] An autonomous driving system is a comprehensive system that integrates numerous advanced technologies. Key components, such as environmental information acquisition and intelligent decision-making and control, rely on innovations and breakthroughs in sensor technology, image recognition, electronics, computing, and control. The continued development of autonomous vehicles relies on breakthroughs and innovations in these diverse technologies.

[0102] Key technologies related to autonomous driving systems include environmental perception, logical reasoning and decision-making, motion control, and processor performance. With advances in machine vision (such as 3D camera technology), pattern recognition software (such as optical character recognition programs), and lidar systems (which incorporate global positioning technology and spatial data), onboard computers can control the vehicle's driving by combining machine vision, sensor data, and spatial data.

[0103] Scale: The scale space of a signal refers to the low-frequency signal obtained by filtering the original signal through a series of single-parameter, Gaussian filters of increasing width. The scale space of an image, on the other hand, refers to the image data as the original signal. When a machine vision system analyzes an unknown scene, the computer has no prior knowledge of the scale of objects in the image. It must simultaneously consider the image's description at multiple scales to determine the optimal scale of the object of interest. Typically, an image is constructed as a set of images of different scales, and features of interest are detected at each scale. For example, when using Haar features to detect faces, because the computer does not know the dimensions of the face in the image, it is necessary to generate an image pyramid consisting of a series of images of different sizes, scanning each image to find possible faces. Image pyramiding can efficiently represent images at multiple scales. The general steps of image pyramiding include: first, smoothing the image with a low-pass filter. Then, the smoothed image is downsampled (typically by 1 / 2 in both the horizontal and vertical directions) to produce a series of downsampled images. The term "scale" involved in the embodiments of the present disclosure refers to the scale of the image frame in the scale space. In other words, it also specifically refers to the resolution (or size) of the image frame, where the resolution of the image frame is usually positively correlated with the size. The higher the resolution, the larger the image frame size, and the lower the resolution, the smaller the image frame size.

[0104] Resolution: The "resolution" involved in the embodiments of the present disclosure refers to image resolution, that is, the amount of information stored in the image, which is the number of pixels per inch in the image. The unit of resolution is PPI (Pixels Per Inch). Image resolution is used to indicate the clarity of the image. A larger resolution represents a higher image clarity, and a smaller resolution represents a lower image clarity. The image resolution is determined by the parameters of the sensing device (such as a camera) that captures the image.

[0105] Scene flow analysis: Scene flow analysis is also known as scene flow estimation. The movement of targets (physical objects such as people, animals, obstacles, and vehicles) and scenes (physical environments such as buildings, roads, and traffic lights) in three-dimensional space is called scene flow. Although scene flow itself is movement in three-dimensional space, after being captured by a camera, the projection of the scene flow on the two-dimensional image plane also forms movement. This flow, which is manifested as the brightness pattern of the image plane, is called optical flow. The main purpose of scene flow analysis is to accurately estimate movement in 3D (three-dimensional) scenes; the main purpose of optical flow analysis is to reliably, quickly, accurately, and robustly estimate movement in 2D (two-dimensional) images. Scene flow analysis is an important method for moving target detection and analysis, and can be applied to many technical fields such as intelligent driving, robot navigation, medical image processing, traffic control, and intelligent video surveillance.

[0106] With the development of computer and mobile communication technologies, autonomous vehicles are gaining widespread attention as an emerging technology. Autonomous vehicles, also known as unmanned vehicles or wheeled mobile robots, are intelligent vehicles controlled by an autonomous driving system.

[0107] In autonomous driving systems, (on-board) cameras are an important sensor. They can capture video streams of the vehicle's surroundings (equivalent to projecting the scene stream onto a two-dimensional image frame plane). By performing scene flow analysis (including optical flow analysis and depth prediction) on the video stream, they can be used to perceive the movement of objects outside the vehicle, especially obstacles, so as to establish autonomous driving strategies and avoid possible driving hazards of unmanned vehicles.

[0108] In an embodiment of the present application, an image processing method is provided that can combine optical flow analysis and depth detection to achieve scene flow analysis, and also combines scene flow analysis with deep learning, that is, using a deep learning model to perform optical flow analysis and depth detection, which is a new breakthrough in the field of computer vision. Since deep learning has a strong feature processing capability, it is possible to find the two most similar related pixels in adjacent image frames by calculating the feature similarity between adjacent image frames, thereby representing the movement of the same pixel between adjacent image frames. Depth detection refers to predicting the depth or distance of adjacent image frames relative to the observation point (i.e., camera) by a deep learning method, so as to obtain the difference in the depth dimension. Through the above-mentioned deep learning method, a new method for predicting the movement of objects (including entities such as objects and human bodies) in continuous image frames can be provided, that is, each image frame can output optical flow detection information and depth detection information, and the object's motion speed in three-dimensional space can be calculated based on the optical flow detection information and depth detection information. In addition, a simple, easy-to-use and portable method for calculating the correlation matrix is provided, which can improve the calculation efficiency of the correlation matrix and enhance the versatility of the above-mentioned method.

[0109] Figure 1 This is a schematic diagram of an implementation environment of an image processing method provided in an embodiment of the present application. Figure 1 , in this implementation environment, it includes a vehicle terminal 101 and a server 102.

[0110] The vehicle-mounted terminal 101 is used to control the vehicle's route and driving parameters. The vehicle-mounted terminal 101 is bound to the vehicle, which includes traditional cars, smart cars, unmanned vehicles, electric cars, bicycles, motorcycles and other means of transportation. The vehicle can be manually driven by the driver or driven by an automatic driving system to achieve unmanned driving. Optionally, the vehicle is equipped with functional modules such as on-board sensors, positioning components, (on-board) cameras, controllers, and data processors. The above functional modules can use modern mobile communication and network technologies such as the Internet of Vehicles, 5G (5th Generation Mobile Networks, fifth-generation mobile communication technology) and V2X (Vehicle To X, vehicle-to-vehicle wireless communication technology) to achieve interchange and sharing between traffic participants, thereby having functions such as sensing perception, decision-making planning, and control execution in complex environments.

[0111] In an exemplary scenario, taking an unmanned vehicle as an example, the onboard terminal of the unmanned vehicle is configured with an automatic driving system, and a connection is established with the server 102 through the automatic driving system to access all vehicle dynamic information provided by the Internet of Vehicles through the server 102.

[0112] The Internet of Vehicles (IoV), or the Vehicle Internet of Things (IoT), uses moving vehicles as information sensors. Leveraging next-generation information and communication technologies, it enables network connectivity between vehicles, people, roads, service platforms, and other entities. This improves the overall intelligent driving capabilities of vehicles, providing users with a safe, comfortable, intelligent, and efficient driving experience and transportation services, while also enhancing traffic efficiency and the intelligence of social transportation services. Optionally, the vehicle-mounted terminal 101 utilizes wireless communication technology to effectively utilize all vehicle dynamic information on the information network platform provided by the server 102, providing various functional services during vehicle operation. IoV generally exhibits the following characteristics: IoV can ensure inter-vehicle spacing, reducing the likelihood of collisions; IoV can assist drivers with real-time navigation; and through communication with other vehicles and network systems, it improves traffic efficiency.

[0113] The vehicle-mounted terminal 101 and the server 102 can be directly or indirectly connected via wired or wireless communication. For example, the vehicle-mounted terminal 101 and the server 102 are wirelessly connected via a vehicle network, which is not specifically limited in this embodiment of the present application.

[0114] Server 102 is used to provide vehicle dynamic information to in-vehicle terminals connected to the information network (i.e., the Internet of Vehicles) platform. Server 102 may comprise at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, server 102 performs primary computing tasks, while in-vehicle terminal 101 performs secondary computing tasks; server 102 performs secondary computing tasks, while in-vehicle terminal 101 performs primary computing tasks; or, alternatively, in-vehicle terminal 101 and server 102 may collaborate on computing tasks using a distributed computing architecture.

[0115] Optionally, the vehicle terminal 101 generally refers to one of multiple vehicle terminals, and the vehicle terminal 101 is configured with an automatic driving system, which can plan the vehicle's driving route and driving parameters based on the vehicle dynamic information provided by the server 102.

[0116] In an exemplary scenario, the vehicle-mounted terminal 101 collects a video stream through a camera, and uses the image processing method provided in an embodiment of the present application to obtain optical flow detection information and depth detection information of each continuous image frame in the video stream, thereby realizing scene flow analysis of external objects. Furthermore, based on the optical flow detection information and depth detection information, the movement speed of external objects can be predicted, thereby guiding the automatic driving system to configure the driving route and driving parameters, and reducing the probability of vehicle danger.

[0117] The types of the vehicle terminal 101 include but are not limited to: at least one of a vehicle console, a smart phone, a tablet computer, a smart watch, a smart speaker, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, a laptop computer or a desktop computer.

[0118] Those skilled in the art will appreciate that the number of vehicle-mounted terminals 101 may be greater or lesser. For example, there may be only one vehicle-mounted terminal 101, or there may be dozens, hundreds, or even more vehicle-mounted terminals 101. The embodiment of the present application does not specifically limit the number and device type of the vehicle-mounted terminals 101.

[0119] Figure 2 This is a flow chart of an image processing method provided by an embodiment of the present application. Figure 2 , this embodiment is executed by an electronic device and includes the following steps:

[0120] 201. An electronic device obtains a plurality of image frames captured by a target camera, where the plurality of image frames have a temporal correlation relationship.

[0121] 202. The electronic device downsamples the multiple image frames respectively to obtain multiple corresponding image sets, each image set including a corresponding image frame and an image obtained by downsampling the image frame.

[0122] 203. The electronic device obtains optical flow detection information and depth detection information of any image frame among the multiple image frames based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame, where the optical flow detection information is used to characterize the movement of the same object between the image frame and the previous frame, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera.

[0123] The method provided in the embodiment of the present application obtains an image set consisting of images in a series of scale spaces by downsampling the image frames captured by the target camera, and uses the images in different scale spaces in the image set to comprehensively predict the optical flow detection information and depth detection information of the current image frame. The hierarchical pyramid features in the image set can be used to make the details contained in the optical flow detection information and the depth detection information richer, thereby improving the accuracy of the optical flow detection information and the depth detection information, that is, improving the accuracy of the scene flow analysis.

[0124] In one possible embodiment, each image set of the multiple image sets includes a corresponding image frame and multiple downsampled images obtained by downsampling the image frame at different sampling multiples, wherein the size of the downsampled image is negatively correlated with the sampling multiple, and the size of the downsampled image is smaller than the size of the image frame.

[0125] In one possible implementation, obtaining the optical flow detection information and the depth detection information of the image frame based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame includes:

[0126] Determining, from an image set corresponding to the image frame and an image set corresponding to a previous frame of the image frame, a plurality of downsampled image pairs corresponding to respective sampling magnifications;

[0127] Based on each of the downsampled image pairs, obtaining intermediate optical flow detection information and intermediate depth detection information;

[0128] Based on the intermediate optical flow detection information and the intermediate depth detection information, optical flow detection information and depth detection information of the image frame are obtained.

[0129] In one possible implementation, obtaining intermediate optical flow detection information and intermediate depth detection information based on each downsampled image pair includes:

[0130] For any sampling magnification, obtaining a first down-sampled image corresponding to the sampling magnification from the image set corresponding to the image frame;

[0131] Obtaining a second down-sampled image corresponding to the sampling magnification from an image set corresponding to a previous frame of the image frame;

[0132] Obtaining a correlation matrix based on the first down-sampled image and the second down-sampled image, where the correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous frame of the image frame at the sampling magnification;

[0133] Based on the correlation matrix, the intermediate optical flow detection information and the intermediate depth detection information are obtained.

[0134] In one possible implementation, the target camera is a binocular camera, the image frame includes a left-eye image and a right-eye image, the first downsampled image includes a first left-eye downsampled image and a first right-eye downsampled image, and the second downsampled image includes a second left-eye downsampled image and a second right-eye downsampled image;

[0135] Based on the first down-sampled image and the second down-sampled image, obtaining a correlation matrix includes:

[0136] Obtaining a first correlation matrix based on the first left-eye downsampled image and the second left-eye downsampled image, wherein the first correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous left-eye image at the sampling magnification;

[0137] Obtaining a second correlation matrix based on the first left-eye downsampled image and the first right-eye downsampled image, where the second correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of the image frame at the sampling magnification;

[0138] Obtaining a third correlation matrix based on the second left-eye downsampled image and the second right-eye downsampled image, where the third correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of a previous frame of the image frame at the sampling magnification;

[0139] A fourth correlation matrix is obtained based on the second correlation matrix and the third correlation matrix. The fourth correlation matrix is used to represent the degree of parallax correlation between the image frame and a previous frame of the image frame at the sampling magnification.

[0140] In one possible implementation, obtaining a correlation matrix based on the first down-sampled image and the second down-sampled image includes:

[0141] For each pixel in the first down-sampled image, determining a sliding window including the pixel and a plurality of neighboring pixels of the pixel;

[0142] Based on the target step size, sliding sampling is performed on the second down-sampled image according to the sliding window to obtain a correlation coefficient between the pixel and each pixel in the second down-sampled image.

[0143] In a possible implementation, performing sliding sampling on the second downsampled image according to the sliding window to obtain a correlation coefficient between the pixel and each pixel in the second downsampled image includes:

[0144] Determine, from the first down-sampled image, a first vector consisting of the pixel and a plurality of neighboring pixels of the pixel;

[0145] For any sliding sampling, determining, from the second down-sampled image, a second vector consisting of a target pixel contained in the sliding window and a plurality of neighboring pixels of the target pixel;

[0146] A dot product between the first vector and the second vector is determined as a correlation coefficient between the pixel and the target pixel.

[0147] In one possible implementation, obtaining the optical flow detection information and the depth detection information of the image frame based on the intermediate optical flow detection information and the intermediate depth detection information includes:

[0148] Acquire a third down-sampled image based on the intermediate optical flow detection information, the intermediate depth detection information, and the second down-sampled image, where the third down-sampled image is a down-sampled image corresponding to the sampling magnification obtained by prediction based on a previous frame of the image frame;

[0149] Upsampling the third downsampled image to obtain an upsampled image corresponding to a target sampling ratio, where the target sampling ratio is smaller than and closest to the sampling ratio among the multiple sampling ratios;

[0150] Iteratively executing the step of obtaining intermediate optical flow detection information and intermediate depth detection information for the upsampled image and the downsampled image corresponding to the target sampling magnification in the image set corresponding to the image frame;

[0151] During any iteration, when the size of the obtained up-sampled image is the same as the size of the image frame, the optical flow detection information and the depth detection information obtained based on the up-sampled image and the image frame are output.

[0152] In one possible implementation, downsampling the multiple image frames respectively to obtain corresponding multiple image sets includes:

[0153] Inputting each of the multiple image frames into a plurality of convolutional layers connected in series, performing convolution processing on the image frame through the plurality of convolutional layers to obtain a plurality of downsampled images, wherein each downsampled image is a feature map output by a convolutional layer;

[0154] The image frame and the multiple down-sampled images are determined as an image set corresponding to the image frame.

[0155] In a possible implementation, the target camera is a vehicle-mounted camera, and the multiple image frames are multiple consecutive video frames in a video stream recorded by the vehicle-mounted camera.

[0156] All the above optional technical solutions can be combined in any way to form optional embodiments of the present disclosure, and will not be described in detail here.

[0157] Figure 3 This is a flow chart of an image processing method provided by an embodiment of the present application. Figure 3 This embodiment is executed by an electronic device, which is a terminal or server in the above-mentioned implementation environment. Optionally, after the terminal side collects each image frame, scene flow analysis is performed. Optionally, after the terminal side collects each image frame, each image frame is sent to the server. After the server performs scene flow analysis on each image frame, it returns optical flow detection information and depth detection information to the terminal. In the embodiment of the present application, the electronic device is used as an example for description. This embodiment includes the following steps:

[0158] 301. A terminal obtains a plurality of image frames captured by a target camera, where the plurality of image frames have a temporal correlation relationship.

[0159] A terminal refers to any electronic device that supports scene flow analysis of image frames with time-series correlation. The device types of the terminal include: vehicle-mounted terminals, smart phones, tablet computers, smart watches, smart speakers, laptop computers, desktop computers, etc. The embodiments of this application do not specifically limit the device type of the terminal.

[0160] Optionally, the target camera refers to any camera installed on the terminal, and the target camera includes a monocular camera or a binocular camera. The embodiment of the present application supports scene flow analysis of both monocular images and binocular images.

[0161] Optionally, the target camera is not installed on this terminal, but is a camera installed on an acquisition device. After the acquisition device acquires the multiple image frames through the camera, the multiple image frames are sent to the terminal for scene flow analysis. That is, the embodiment of the present application does not specifically limit whether the target camera is installed on the terminal that processes the image.

[0162] In some embodiments, the terminal responds to a shooting instruction and calls the target camera through an API (Application Programming Interface) to acquire a video stream, and obtains the multiple image frames based on a video frame sequence in the video stream in chronological order of acquisition time. For example, all video frames in the video frame sequence are determined as the multiple image frames. For another example, one video frame is sampled as an image frame at every preset number of intervals in the video frame sequence, and the multiple image frames are finally obtained after multiple consecutive samplings. For another example, each key frame in the video frame sequence is determined as the multiple image frames. For another example, object recognition (i.e., target detection, detecting whether the video frame contains a target object, for example, the target object is an obstacle) is performed on each video frame in the video frame sequence, and each video frame containing the target object is determined as the multiple image frames. The embodiments of the present application do not specifically limit the method for acquiring the multiple image frames.

[0163] In the above process, the multiple image frames must have a temporal correlation (have temporal correlation) and can be regarded as a time series. Only when the multiple image frames are guaranteed to have temporal correlation can scene flow analysis be performed, that is, by performing scene flow analysis on any two adjacent image frames, and so on, the movement conditions of the objects contained in the image frames within a time period (such as movement direction, movement rate, etc.) can be obtained.

[0164] In an exemplary scenario, the terminal is a vehicle-mounted terminal, and the target camera is a vehicle-mounted camera. Then, the multiple image frames are multiple consecutive video frames in the video stream recorded by the vehicle-mounted camera. For example, recording starts at time T=0, and the current time is time T=t. Then, the multiple image frames in the video stream recorded by the vehicle-mounted camera are represented as: [I0, I1, ..., I t-1 , I t ].

[0165] 302. The terminal downsamples the multiple image frames respectively to obtain multiple corresponding image sets, each of which includes a corresponding image frame and multiple downsampled images obtained by downsampling the image frame at multiple sampling rates.

[0166] Each image set in the multiple image sets includes a corresponding image frame and multiple downsampled images obtained by downsampling the image frame at different sampling multiples, wherein the size of the downsampled image is negatively correlated with the sampling multiple, that is, the larger the sampling multiple, the smaller the size of the downsampled image, and the smaller the sampling multiple, the larger the size of the downsampled image, and since the resolution will be reduced during the downsampling process, the size of the downsampled image is smaller than the size of the image frame.

[0167] In some embodiments, the multiple sampling magnifications refer to a series of magnifications arranged from small to large. For example, each sampling magnification is twice the previous sampling magnification. For example, the multiple sampling magnifications are: 2, 4, 8, 16..., at this time, the down-sampled image with a sampling magnification of 2 is the image obtained by downsampling the original image frame by 2 times, and the down-sampled image with a sampling magnification of 4 is the image obtained by downsampling the down-sampled image with a sampling magnification of 2 by 2 times. And so on, a series of down-sampled images with gradually reduced sizes can be obtained hierarchically and progressively, and the original image frame and the down-sampled images are determined as the image set corresponding to the image frame. This image set also constitutes a series of features of different scales of the image frame in the scale space. Since the size of the image tends to decrease step by step during the downsampling process, which is similar to a pyramid structure, the image set is also called the pyramid feature of the corresponding image frame.

[0168] Optionally, when obtaining the corresponding image set for each image frame, a pyramid feature extraction algorithm such as a Gaussian pyramid, a Laplacian pyramid, or a DOG pyramid (Difference of Gaussian) is adopted, or convolution, pooling, or the like can be adopted for step-by-step downsampling. This embodiment of the present application does not specifically limit this.

[0169] In some embodiments, downsampling is performed using a convolutional approach as an example. For each of the multiple image frames, the image frame is input into multiple convolutional layers connected in series. The multiple convolutional layers perform convolution processing on the image frame to obtain multiple downsampled images, each of which is a feature map output by a convolutional layer. The image frame and the multiple downsampled images are then determined as an image set corresponding to the image frame. The number of the multiple convolutional layers can be a value greater than or equal to 1, for example, 6.

[0170] In the above process, by downsampling in a convolutional manner, it is possible to change the image size and increase the number of channels of the resulting feature map (i.e., the downsampled image). That is, deep pyramid features are also extracted during the downsampling process. Since the images downsampled based on the image frame are all feature maps output by each convolutional layer, the expressive power of the corresponding image frames included in each image set and the images downsampled based on the image frame is improved.

[0171] Figure 4 This is a schematic diagram of the principle of a convolution operation provided by an embodiment of the present application, such as Figure 4As shown, the left part is the original image frame 401, and the right part is the feature map obtained by convolution, namely the downsampled image 402. The first pixel "4" in the downsampled image 402 is the value obtained by element-wise multiplication of a 3×3 pixel matrix in the upper left corner of the original image frame 401 and a 3×3 convolution kernel, and then the sum is obtained. That is, the number in the middle of the shaded square in the image frame 401 represents the pixel value, and the number in the subscript of the shaded square represents the corresponding convolution kernel coefficient. The number in the middle of each shaded square is multiplied by the number in the subscript, and the values obtained by multiplying the shaded squares are summed to obtain the first pixel "4" in the downsampled image 402. In other words, 4 = (1×1) + (1×0) + (1×1) + (0×0) + (1×1) + (1×0) + (0×1) + (0×0) + (1×1).

[0172] 303. The terminal obtains, for any image frame among the multiple image frames, a first down-sampled image corresponding to the sampling rate from an image set corresponding to the image frame, starting from the maximum sampling rate.

[0173] In some embodiments, for each image frame, the optical flow detection information and depth detection information of the image frame can be finally obtained based on the image set corresponding to the current image frame and the image set corresponding to the previous frame of the current image frame, wherein the optical flow detection information is used to characterize the movement of the same object between the image frame and the previous frame, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera. Therefore, in the embodiments of the present application, taking any image frame as an example, the method of obtaining the optical flow detection information and depth detection information of the current image frame is introduced.

[0174] For example, for multiple image frames [I0, I1, ..., I t-1 , I t ], each image frame is downsampled 6 times step by step to obtain 6 downsampled images (for example, it is input into 6 convolutional layers, and each convolutional layer outputs a feature map, i.e., a downsampled image). Then, each image frame and the corresponding 6 downsampled images can constitute the image set of this image frame. t Take scene flow analysis as an example, the image frame I at the current moment t The image collection includes: t , I t 1 , I t 2 , I t 3 , I t 4 , I t 5 , I t6}, where I t The superscript indicates the scale layer number of the corresponding downsampled image in the image set. The larger the scale layer number, the larger the sampling rate and the smaller the size of the corresponding downsampled image.

[0175] In the embodiment of the present application, steps 303-308 are an iterative execution cycle. The first iteration of this iterative process starts from the downsampled image corresponding to the maximum sampling magnification, that is, the first iteration starts from the downsampled image with the smallest size. Since the image frame I at the current moment needs to be referenced during scene flow analysis, t and the image frame at the previous moment (i.e., the previous frame) I t-1 The pixels of each image are also in the same scale space (i.e. the scale is the same, which means the size and sampling rate are the same). Therefore, in order to distinguish them, the image frame I at the current moment is t The downsampled image in the image set is called the first downsampled image. t-1 The downsampled image in the image set is called the second downsampled image.

[0176] 304. The terminal obtains a second down-sampled image corresponding to the sampling magnification from an image set corresponding to a previous frame of the image frame.

[0177] The above step 304 is: the terminal selects the second down-sampled image corresponding to the same sampling ratio as in the above step 303 from the image set of the previous frame. The selection method is similar to the above step 303 and will not be repeated here.

[0178] In an exemplary scenario, when downsampling is performed 6 times, the image frame I at the current moment is t The downsampled image {I t 1 , I t 2 , I t 3 , I t 4 , I t 5 , I t 6} is called the first down-sampled image, the previous frame I t-1 The downsampled image {I t-1 1 , I t-1 2 , I t-1 3 , I t-1 4 , I t-1 5 , It-1 6} is called the second down-sampled image. In the first iteration, the image frame I at the current moment is obtained by the above step 303. t In the image set (t-th frame), the first down-sampled image I corresponding to the largest sampling rate is selected. t 6 , through the above step 304 from the previous frame I t-1 In the image set, the second down-sampled image I corresponding to the largest sampling ratio is selected t-1 6 .

[0179] In the above steps 303-304, since a first downsampled image of the current frame and a second downsampled image of the previous frame are obtained in each iteration, the two can constitute a downsampled image pair with the same sampling ratio (the same size). Therefore, in multiple iterations, it is equivalent to determining multiple downsampled image pairs corresponding to their respective sampling ratios from the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame.

[0180] 305. The terminal obtains a correlation matrix based on the first down-sampled image and the second down-sampled image. The correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous frame of the image frame at the sampling magnification.

[0181] When obtaining the correlation matrix, since the target camera can be a monocular camera or a binocular camera, the embodiments of this application will discuss the monocular case and the binocular case separately. Next, the method of obtaining the correlation matrix in the monocular case is introduced. In the monocular case, the target camera captures a single image frame at each moment (equivalent to considering only the left eye image), so it is only necessary to calculate a correlation matrix based on the first downsampled image and the second downsampled image.

[0182] In some embodiments, the terminal obtains the correlation matrix in the following manner: for each pixel in the first downsampled image, determine a sliding window containing the pixel and multiple neighboring pixels of the pixel; based on the target step size, perform sliding sampling on the second downsampled image according to the sliding window to obtain the correlation coefficient between the pixel and each pixel in the second downsampled image.

[0183] In the above process, traditional scene flow analysis uses a calculation strategy based on a correlation matrix encapsulated in a dynamic link library. However, this reliance on the dynamic link library results in slow and inefficient calculations, and often fails to operate due to improper environment or installation. However, through the aforementioned sliding sampling method, the correlation coefficient between each pixel in the first downsampled image and each pixel in the second downsampled image is calculated. The two pixels with the largest correlation coefficients represent the pixels most likely to have moved from the previous frame to the current frame, thereby reflecting the degree of pixel correlation between the current and previous frames.

[0184] Optionally, for each pixel in the first downsampled image, the terminal determines multiple neighboring pixels that meet the preset sliding window size based on the pixel as the center point of the sliding window, thereby obtaining a sliding window containing the pixel and multiple neighboring pixels of the pixel.

[0185] Optionally, when performing sliding sampling, the sliding window in the first down-sampled image is first fixed unchanged, and from the first down-sampled image, a first vector consisting of the pixel and multiple neighboring pixels of the pixel can be determined (the first vector does not change during sliding sampling). Then, a sliding window of the same size is used to move according to the target step size in the second down-sampled image, for example, starting from the upper left corner or from the lower right corner. The embodiment of the present application does not specifically limit the starting point of the sliding sampling, as long as the sliding window traverses each pixel in the second down-sampled image once.

[0186] Optionally, for any sliding sampling, the terminal determines from the second down-sampled image a second vector consisting of a target pixel contained in the sliding window and multiple neighboring pixels of the target pixel, wherein the target pixel refers to the pixel at the center point of the position of the sliding window in the second down-sampled image, and the method for determining the neighboring pixels of the target pixel is similar to the method for determining the domain pixels of the pixel, and the method for determining the second vector is also similar to the method for determining the first vector. The embodiment of the present application does not make any specific limitations on this.

[0187] Optionally, after obtaining the first vector and the second vector, the dot product between the first vector and the second vector is determined as the correlation coefficient between the pixel and the target pixel. Since the more similar the two vectors are, the closer the dot product of the two vectors is to 1, and the more dissimilar the two vectors are, the closer the dot product of the two vectors is to 0, therefore, by calculating the dot product between the first vector determined by the pixel and its neighboring pixels and the second vector determined by the target pixel and its neighboring pixels, it can be reflected whether the first vector and the second vector are similar. When the first vector and the second vector are more similar, it is more likely that they are associated pixels generated by the motion of the same object in the current frame and the previous frame (that is, it reflects the degree of association between any two pixels). Therefore, the dot product of the first vector and the second vector can be used as the correlation coefficient.

[0188] For each pixel in the first down-sampled image, a series of correlation coefficients can be obtained, and finally all the correlation coefficients of all the pixels in the first down-sampled image can form a correlation matrix.

[0189] In some embodiments, when performing the above-mentioned sliding sampling, in order to fully utilize the edge pixel information of the second down-sampled image, the second down-sampled image is expanded by performing zero padding, that is, a circle of pixels with a value of 0 is padded around the edge pixels of the second down-sampled image, and then the above-mentioned sliding sampling is performed on the zero-padded second down-sampled image, so that each edge pixel can participate in the sliding sampling with the pixels with a value of 0 padded in the outer circle as a neighboring pixel. This is because if zero padding is not performed, the edge pixel cannot be used as the center point of the sliding window for sliding sampling, and therefore a part of the detail information of the edge pixel will be lost. The zero padding method enables the edge pixel to be used as the center point of the sliding window for sliding sampling, thereby improving the accuracy of the correlation matrix finally obtained.

[0190] Figure 5 This is a schematic diagram of the principle of a zero-filling method provided in an embodiment of the present application, such as Figure 5 As shown, a method for calculating the correlation coefficient in the two-dimensional direction is shown. At this time, it is assumed that the number of channels of the second down-sampled image is 1 (that is, the second down-sampled image is a single-channel image, such as a black and white image). The left part represents the original second down-sampled image 501, and the right part represents the second down-sampled image 502 obtained by filling the edge pixels of the second down-sampled image 501 with a circle of pixels with a value of 0. Schematically, a sliding window with a size of 2×2 is used to perform sliding sampling on the second down-sampled image 502 after zero filling with a target step size of 1, and finally a series of correlation coefficients can be calculated, and a correlation matrix can be constructed.

[0191] Figure 6This is a schematic diagram of the principle of a zero-filling method provided in an embodiment of the present application, such as Figure 6 As shown, the calculation method of the correlation coefficient in the three-dimensional direction is shown. At this time, it is assumed that the number of channels of the second down-sampled image is n, n>1. At this time, since the original image frame is usually three channels of red, green and blue (RGB), or four channels of RGB+α (transparency), after down-sampling by convolution, although the image size will become smaller, the number of image channels will become larger, that is, the convolution method extracts a high-level depth effect. In this case, each channel in the multi-channel second down-sampled image can be regarded as a single-channel image, and each channel can be used Figure 5 Regarding the two-dimensional correlation calculation method shown, it should be noted that when the original second down-sampled image 601 is padded with zeros, the edge pixels of each channel need to be padded with a circle of pixels with a value of 0, and finally the second down-sampled image 602 after the zero-padded image is obtained. Moreover, when performing sliding sampling, it is necessary to traverse all pixels in all channels in the second down-sampled image 602 after the zero-padded image, and finally a series of richer correlation coefficients can be calculated, and a correlation matrix can be constructed.

[0192] The above process describes how to obtain a correlation matrix for a monocular camera. In a binocular camera, the target camera is a binocular camera. In practice, each image frame captured by the binocular camera includes a left-eye image and a right-eye image. When downsampling each image frame, both the left and right images need to be downsampled. That is, the downsampled images obtained by downsampling each image frame include a left-eye downsampled image and a right-eye downsampled image. Based on this, it is clear that the first downsampled image obtained in step 303 includes a first left-eye downsampled image and a first right-eye downsampled image, and the second downsampled image obtained in step 304 includes a second left-eye downsampled image and a second right-eye downsampled image.

[0193] Therefore, when calculating the correlation matrix, it is necessary to use a total of four images, namely, the first left-eye downsampled image, the first right-eye downsampled image, the second left-eye downsampled image, and the second right-eye downsampled image, to calculate multiple correlation matrices.

[0194] Optionally, the terminal obtains a first correlation matrix based on the first left-eye downsampled image and the second left-eye downsampled image, and the first correlation matrix is used to characterize the degree of pixel correlation between the left-eye image of the image frame and the previous frame of the image frame at the sampling magnification; obtains a second correlation matrix based on the first left-eye downsampled image and the first right-eye downsampled image, and the second correlation matrix is used to characterize the disparity of the left-eye image and the right-eye image of the image frame at the sampling magnification; obtains a third correlation matrix based on the second left-eye downsampled image and the second right-eye downsampled image, and the third correlation matrix is used to characterize the disparity of the left-eye image and the right-eye image of the previous frame of the image frame at the sampling magnification; obtains a fourth correlation matrix based on the second correlation matrix and the third correlation matrix, and the fourth correlation matrix is used to characterize the degree of disparity correlation between the image frame and the previous frame of the image frame at the sampling magnification.

[0195] The calculation methods of the above-mentioned first correlation matrix, second correlation matrix, third correlation matrix and fourth correlation matrix are similar to the calculation method of a single correlation matrix in the monocular case, and are not described in detail here.

[0196] In the above process, by introducing the monocular and binocular situations respectively, the image processing method provided in the embodiment of the present application can be applied to different situations such as monocular cameras and binocular cameras, and has high portability and universality. In addition, since the calculation of the correlation matrix does not need to rely on the fixed strategy encapsulated in the dynamic link library or static link library, but can be implemented in the form of a function, the function can be used directly without relying on a specific environment. There is no problem of version mismatch or inapplicability. It can improve the calculation efficiency of the correlation matrix and improve the accuracy of the final predicted optical flow detection information and depth detection information. It is also very versatile and portable.

[0197] 306. The terminal obtains intermediate optical flow detection information and intermediate depth detection information based on the correlation matrix.

[0198] Among them, the intermediate optical flow detection information is used to characterize the movement of the same object between the image frame and the previous frame at the sampling rate, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera at the sampling rate.

[0199] In some embodiments, the terminal inputs the correlation matrix into an estimator, and maps the correlation matrix through the estimator to obtain the intermediate optical flow detection information and the intermediate depth detection information. For example, for each pixel in the first down-sampled image, based on the correlation matrix, the first pixel with the largest correlation coefficient with the pixel is determined, so that the pixel and the first pixel are regarded as a pair of associated pixels with a motion relationship. For example, for the target object in the three-dimensional space perceived by the target camera (including entities such as human bodies and objects, such as obstacles, vehicles, pedestrians, etc.), the three-dimensional space point on the target object at time T=t-1 is projected to the first pixel in the t-1 frame (previous frame). Since the target object moves in the three-dimensional space, the same three-dimensional space point on the target object at time T=t is projected to the pixel in the t-1 frame (current frame). Based on each pixel in the first downsampled image and the corresponding first pixel, an intermediate optical flow detection information can be determined. The intermediate optical flow detection information reflects the optical flow detection information under the sampling magnification of this iteration, but it is necessary to gradually reduce the sampling magnification and enlarge the image size through multiple iterations until the final optical flow detection information in the scale space is output in the same scale space as the original image frame size.

[0200] Schematically, the intermediate optical flow detection information is an optical flow detection map. Each pixel in the optical flow detection map represents the direction and amount of movement of the pixel at the corresponding position in the first downsampled image, that is, the direction of movement from the first pixel to the pixel and the specific amount of movement in the direction. Figure 7 is a standard optical flow parameter map provided in the embodiment of the present application, such as Figure 7 As shown, Figure 7 Pixels in different directions in the first down-sampled image are colored with different colors in the color spectrum, so that for each pixel in the first down-sampled image, after determining the first pixel with the largest correlation coefficient in the second down-sampled image, based on the displacement direction and displacement distance of the displacement vector determined from the first pixel to the pixel, Figure 7 The standard optical flow parameter map is sampled, and the color value of the corresponding sampling point in the standard optical flow parameter map is assigned to the pixel at the corresponding position in the optical flow detection map. For example, if the displacement direction from the first pixel to the pixel is the upper left, then the color value of the sampling point corresponding to the upper left of the standard optical flow parameter map is sampled (for example, light blue), and the color value of the sampling point is assigned to the pixel at the corresponding position in the optical flow detection map, that is, the pixel at the corresponding position in the final optical flow detection map is light blue. It should be noted that Figure 4This is only an exemplary description of a standard optical flow parameter map. Different visualization methods can be configured for the standard optical flow parameter map. Under different visualization methods, even if the movement is in the same direction, the sampled color values will be different. The embodiments of the present application do not specifically limit this.

[0201] In some embodiments, based on each pixel in the first downsampled image and the corresponding first pixel, in addition to obtaining intermediate optical flow detection information, intermediate depth detection information can also be obtained. For example, the intermediate depth detection information is a depth detection map. Optionally, the brightness change value between the pixel and the first pixel can reflect the depth change, that is, the distance change, of the target object relative to the observation point (that is, the target camera). Therefore, the brightness change value between each pixel and the corresponding first pixel is assigned to the pixel corresponding to the position of each pixel in the depth detection map, thereby finally obtaining a depth detection map at the current sampling rate.

[0202] Since steps 305 - 308 are an iterative process, the terminal can obtain intermediate optical flow detection information and intermediate depth detection information based on the downsampled image pairs in each scale space.

[0203] 307. The terminal obtains a third down-sampled image based on the intermediate optical flow detection information, the intermediate depth detection information, and the second down-sampled image. The third down-sampled image is a down-sampled image corresponding to the sampling magnification obtained by predicting a previous frame of the image frame.

[0204] In some embodiments, the terminal performs nonlinear mapping on the second downsampled image based on the intermediate optical flow detection information and the intermediate depth detection information to obtain a third downsampled image. Since the intermediate optical flow detection information reflects the optical flow motion information from the previous frame to the current frame, and the intermediate depth detection information reflects the depth change information from the previous frame to the current frame, based on the optical flow motion information and the depth change information, a third downsampled image of the current frame at the same sampling rate can be predicted based on the second downsampled image. That is, at the same sampling rate, the first downsampled image is the actual downsampled image of the current frame, and the third downsampled image is the downsampled image predicted based on the previous frame. For example, for each pixel in the second downsampled image, the displacement vector indicated by the intermediate optical flow detection information is moved, and the brightness value is changed according to the brightness change value indicated by the intermediate depth detection information. Repeating the above operation can predict the third downsampled image estimated for the current frame (t-th frame). It should be noted that the third downsampled image is predicted during the iterative process, but is not actually captured by the target camera. The above nonlinear mapping process can be regarded as performing an affine transformation on the second down-sampled image to obtain a corresponding third down-sampled image.

[0205] In some embodiments, the terminal inputs the second downsampled image, the intermediate optical flow detection information, and the intermediate depth detection information into an affine transformation (warp) layer, and predicts the third downsampled image through the warp layer. In addition, the multiple convolutional layers (downsampled parts) in the above step 302, the module for calculating the correlation matrix in the above step 305 (considered as a correlation layer), the optical flow analysis and depth detection module (i.e., the estimator) in the above step 306, and the warp layer in this step 307 can be used to construct an overall deep learning model, thereby training the various parts of the deep learning model together, which can simplify the training process. Of course, the above-mentioned parts can also be trained separately, and this embodiment of the present application does not specifically limit this.

[0206] 308. The terminal upsamples the third downsampled image to obtain an upsampled image corresponding to a target sampling ratio, where the target sampling ratio is smaller than and closest to the sampling ratio among the multiple sampling ratios.

[0207] In some embodiments, the terminal upsamples the third downsampled image, and the upsampling method includes but is not limited to: deconvolution (Transposed Convolution, also known as transposed convolution), unpooling, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. The embodiment of the present application does not specifically limit the upsampling method.

[0208] In some embodiments, since the above step 302 shows multiple downsamplings in a convolution manner, when upsampling, the deconvolution layer corresponding to the convolution layer can be used to perform upsampling based on the convolution layer corresponding to the sampling ratio selected in this iteration, so that after the third downsampled image is input into the deconvolution layer, the output upsampled image is exactly the same size as the downsampled image output by the convolution layer corresponding to the target sampling ratio, which is equivalent to the upsampling in each iteration being the inverse process of the downsampling corresponding to the target sampling ratio.

[0209] In some embodiments, a neural network may be used for upsampling. For example, Refine Net may be used to implement the upsampling process of the third downsampled image. Other neural networks capable of upsampling may be obtained. This embodiment of the present application does not specifically limit this.

[0210] For example, in the first iteration, the image frame I at the current moment is obtained through the above step 303. t In the image set (t-th frame), the first down-sampled image I corresponding to the largest sampling rate is selected. t6 , through the above step 304 from the previous frame I t-1 In the image set, the second down-sampled image I corresponding to the largest sampling ratio is selected t-1 6 . For the second downsampled image I t-1 6 , using the intermediate optical flow detection information and the intermediate depth detection information, a current frame I' is generated t 6 (t-th frame) predicts the third down-sampled image, and then, the third down-sampled image I' t 6 Perform deconvolution to obtain an upsampled image I' t 5 , thus switching from scale level 6 to scale level 5, then in the following step 309, the upsampled image I' is used in the scale space of the scale level 5. t 5 And the corresponding downsampled image I of the current frame t 5 , respectively, as the first down-sampled image and the second down-sampled image in the next iteration, and steps 305-308 are iteratively performed. That is, in the first iteration, the down-sampled image of the previous frame at the maximum sampling ratio is used as the second down-sampled image. In each subsequent iteration except the first (i.e., the second and subsequent iterations), the down-sampled image of the current frame at the corresponding sampling ratio is used as the second down-sampled image.

[0211] In the above process, the third down-sampled image predicted in the current scale space is switched to the scale space of the previous layer after upsampling. For the scale space of the previous layer, the comparison between the generated up-sampled image of the current frame and the first down-sampled image of the current frame actually collected can provide a self-supervisory signal to the model to improve the prediction accuracy of the model.

[0212] Figure 8 This is a schematic diagram of the architecture of a scene flow analysis model provided by an embodiment of the present application. Figure 8As shown, the scene flow analysis model is an exemplary illustration of a deep learning model. The scene flow analysis model includes a downsampling module 801, a correlation matching module 802, an optical flow analysis and depth detection module 803, a warp affine transformation module 804 and an upsampling module 805. When the target camera is a binocular camera, it is assumed that the image frame at the current moment is the second frame and the previous frame is the first frame. The input images include: the left eye image L1 and the right eye image R1 of the first frame, and the left eye image L2 and the right eye image R2 of the second frame. A total of 4 images, L1, R1, L2 and R2, are input into the downsampling module 801 for a series of downsampling to obtain 4 image sets corresponding to the 4 images. In any iterative process, in the same scale space, the first correlation matrix is extracted for the downsampled image pair corresponding to L1 and L2, the second correlation matrix is extracted for the downsampled image pair corresponding to L2 and R2, and the third correlation matrix is extracted for the downsampled image pair corresponding to L1 and R1. The second correlation matrix and the third correlation matrix are used to extract the fourth correlation matrix. That is, in the correlation matching module 802, four correlation matrices are calculated and input into the optical flow analysis and depth detection module 803 to extract the intermediate optical flow detection information F2. K and intermediate depth detection information D2 K , where K represents the number of scale layers in the current scale space. Then, the intermediate optical flow detection information F2 K and intermediate depth detection information D2 K The downsampled images corresponding to the original left-eye image L1 and right-eye image R1 of the first frame are input into the warp affine transformation module 804 to predict the left-eye downsampled image and the right-eye downsampled image of the second frame. Then, the left-eye downsampled image and the right-eye downsampled image predicted by the warp affine transformation module 804 are input into the upsampling module 805 to predict a left-eye upsampled image and a right-eye upsampled image, which can be put into a new round of iterative process corresponding to the scale space.

[0213] 309. The terminal iteratively executes steps 305-308 above on the downsampled image corresponding to the target sampling ratio in the image set corresponding to the upsampled image and the image frame until the size of the obtained upsampled image is the same as the size of the image frame, and outputs the optical flow detection information and depth detection information obtained based on the upsampled image and the image frame.

[0214] In the above step 309, the upsampled image is equivalent to the first downsampled image in the next round of iteration, and the downsampled image corresponding to the target sampling ratio in the image set corresponding to the image frame is equivalent to the second downsampled image in the next round of iteration. Therefore, iterative execution of operations similar to the above steps 305-308 can continuously perform upsampling. Since the image size will be enlarged during the upsampling process, it is equivalent to starting from the smallest scale space, iteratively updating the intermediate optical flow detection information and the intermediate depth detection information step by step, and at the same time enlarging the image size to switch to a larger scale space. Therefore, at the highest level, that is, the largest scale space (the scale space corresponding to the size of the original image frame), the intermediate optical flow detection information and intermediate depth detection information obtained are the final optical flow detection information and depth detection information of the image frame.

[0215] In other words, by iteratively executing the steps of obtaining intermediate optical flow detection information and intermediate depth detection information, it is possible to integrate optical flow motion information and depth change information at different scales, and ultimately output final optical flow detection information and depth detection information at the highest scale space where the image frame resides. In other words, based on the intermediate optical flow detection information and intermediate depth detection information obtained at each iteration, it is possible to guide the continuous adjustment and correction of the intermediate optical flow detection information and intermediate depth detection information, thereby ensuring that the optical flow detection information and depth detection information obtained for the image frame have higher accuracy.

[0216] In the above steps 303-309, the guidance is to obtain optical flow detection information and depth detection information of the image frame based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame, wherein the optical flow detection information is used to characterize the motion of the same object between the image frame and the previous frame, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera. Since the optical flow motion information and depth change information contained in the image obtained based on downsampling of the image frame are integrated, the optical flow detection information and depth detection information have higher accuracy.

[0217] All the above optional technical solutions can be combined in any way to form optional embodiments of the present disclosure, and will not be described in detail here.

[0218] The method provided in the embodiment of the present application obtains an image set consisting of images in a series of scale spaces by downsampling the image frames captured by the target camera, and uses the images in different scale spaces in the image set to comprehensively predict the optical flow detection information and depth detection information of the current image frame. The hierarchical pyramid features in the image set can be used to make the details contained in the optical flow detection information and the depth detection information richer, thereby improving the accuracy of the optical flow detection information and the depth detection information, that is, improving the accuracy of the scene flow analysis, and more complex three-dimensional scene flows can be predicted through the optical flow detection information and the depth detection information.

[0219] Figure 9 This is a principle flow chart of an image processing method provided in an embodiment of the present application, such as Figure 9 As shown, this embodiment includes the following processes:

[0220] Step 1: Input multiple continuous image frames.

[0221] Step 2: downsample the multiple image frames to obtain corresponding multiple image sets, each image set is used to represent the pyramid features of the corresponding image frame, that is, hierarchically extract the image pyramid features.

[0222] Step 3: Calculate feature correlation, that is, in a single iteration, calculate the similarity matrix for the downsampled image pairs of adjacent image frames at the level scale corresponding to this iteration.

[0223] Step 4: Matching process, that is, for each pixel in the current frame, find the first pixel with the largest similarity coefficient in the previous frame.

[0224] Step 5: Using the estimator, obtain the intermediate optical flow detection information and the intermediate depth detection information according to the matching results of step 4.

[0225] Step 6. Based on the intermediate optical flow detection information and intermediate depth detection information output in step 5, the affine transformation process of the original downsampled image (second downsampled image) at the hierarchical scale corresponding to this iteration in the layered image pyramid feature in step 2 is guided to generate a new downsampled image (third downsampled image).

[0226] Among them, in the first iteration, the second down-sampled image is the down-sampled image of the scale space corresponding to the previous frame (t-1 frame), and in the second and any subsequent iterations, the second down-sampled image is the down-sampled image of the scale space corresponding to the current frame (t frame).

[0227] Step 7: Upsample the new downsampled image generated in step 6 to obtain an upsampled image.

[0228] Step 8: Calculate the feature correlation between the upsampled image obtained in step 7 and the downsampled image at the previous level in step 2, and then return to step 3.

[0229] Step 9: In the highest level, i.e., the largest scale space, output the optical flow detection information and depth detection information calculated in step 6 in this scale space.

[0230] In the embodiments of the present application, a deep learning framework and depth prediction are combined to predict more complex three-dimensional scene flows and enrich the motion information of pixels. In addition, due to the use of multi-scale spatial image pyramid features for deep learning and the use of a layer structure to increase the receptive field, the interference of similar pixels on optical flow detection information and depth detection information can be reduced, thereby improving the accuracy of optical flow detection information and depth detection information.

[0231] Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Please refer to Figure 10 , the device comprises:

[0232] A first acquisition module 1001 is configured to acquire a plurality of image frames captured by a target camera, wherein the plurality of image frames have a temporal correlation relationship;

[0233] A downsampling module 1002 is configured to downsample the multiple image frames respectively to obtain corresponding multiple image sets, each image set including a corresponding image frame and an image obtained by downsampling the image frame;

[0234] The second acquisition module 1003 is used to obtain optical flow detection information and depth detection information of any image frame among the multiple image frames based on the image set corresponding to the image frame and the image set corresponding to the previous frame of the image frame. The optical flow detection information is used to characterize the movement of the same object between the image frame and the previous frame, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera.

[0235] The device provided in the embodiment of the present application obtains an image set consisting of images in a series of scale spaces by downsampling the image frames captured by the target camera, and uses images in different scale spaces in the image set to comprehensively predict the optical flow detection information and depth detection information of the current image frame. The hierarchical pyramid features in the image set can be used to enrich the details contained in the optical flow detection information and depth detection information, thereby improving the accuracy of the optical flow detection information and depth detection information, that is, improving the accuracy of scene flow analysis.

[0236] In one possible embodiment, each image set of the multiple image sets includes a corresponding image frame and multiple downsampled images obtained by downsampling the image frame at different sampling multiples, wherein the size of the downsampled image is negatively correlated with the sampling multiple, and the size of the downsampled image is smaller than the size of the image frame.

[0237] In one possible implementation, based on Figure 10 The second acquisition module 1003 includes:

[0238] a determination submodule, configured to determine, from an image set corresponding to the image frame and an image set corresponding to a previous frame of the image frame, a plurality of downsampled image pairs corresponding to respective sampling magnifications;

[0239] A first acquisition submodule is configured to acquire intermediate optical flow detection information and intermediate depth detection information based on each downsampled image pair;

[0240] The second acquisition submodule is configured to acquire optical flow detection information and depth detection information of the image frame based on the intermediate optical flow detection information and the intermediate depth detection information.

[0241] In one possible implementation, based on Figure 10 The first acquisition submodule includes:

[0242] A first acquiring unit is configured to acquire, for any sampling magnification, a first down-sampled image corresponding to the sampling magnification from an image set corresponding to the image frame;

[0243] The first acquisition unit is further configured to acquire a second down-sampled image corresponding to the sampling magnification from an image set corresponding to a previous frame of the image frame;

[0244] A second acquisition unit is configured to acquire a correlation matrix based on the first down-sampled image and the second down-sampled image, wherein the correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous frame of the image frame at the sampling magnification;

[0245] The third acquisition unit is configured to acquire the intermediate optical flow detection information and the intermediate depth detection information based on the correlation matrix.

[0246] In one possible implementation, the target camera is a binocular camera, the image frame includes a left-eye image and a right-eye image, the first downsampled image includes a first left-eye downsampled image and a first right-eye downsampled image, and the second downsampled image includes a second left-eye downsampled image and a second right-eye downsampled image;

[0247] The second acquiring unit is used for:

[0248] Obtaining a first correlation matrix based on the first left-eye downsampled image and the second left-eye downsampled image, wherein the first correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous left-eye image at the sampling magnification;

[0249] Obtaining a second correlation matrix based on the first left-eye downsampled image and the first right-eye downsampled image, where the second correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of the image frame at the sampling magnification;

[0250] Obtaining a third correlation matrix based on the second left-eye downsampled image and the second right-eye downsampled image, where the third correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of a previous frame of the image frame at the sampling magnification;

[0251] A fourth correlation matrix is obtained based on the second correlation matrix and the third correlation matrix. The fourth correlation matrix is used to represent the degree of parallax correlation between the image frame and a previous frame of the image frame at the sampling magnification.

[0252] In one possible implementation, based on Figure 10 The second acquisition unit includes:

[0253] a determining subunit, configured to determine, for each pixel in the first down-sampled image, a sliding window including the pixel and a plurality of neighboring pixels of the pixel;

[0254] The sampling subunit is configured to perform sliding sampling on the second downsampled image according to the sliding window based on a target step size, and obtain a correlation coefficient between the pixel and each pixel in the second downsampled image.

[0255] In one possible implementation, the sampling subunit is configured to:

[0256] Determine, from the first down-sampled image, a first vector consisting of the pixel and a plurality of neighboring pixels of the pixel;

[0257] For any sliding sampling, determining, from the second down-sampled image, a second vector consisting of a target pixel contained in the sliding window and a plurality of neighboring pixels of the target pixel;

[0258] A dot product between the first vector and the second vector is determined as a correlation coefficient between the pixel and the target pixel.

[0259] In one possible implementation, the second acquisition submodule is configured to:

[0260] Acquire a third down-sampled image based on the intermediate optical flow detection information, the intermediate depth detection information, and the second down-sampled image, where the third down-sampled image is a down-sampled image corresponding to the sampling magnification obtained by prediction based on a previous frame of the image frame;

[0261] Upsampling the third downsampled image to obtain an upsampled image corresponding to a target sampling ratio, where the target sampling ratio is smaller than and closest to the sampling ratio among the multiple sampling ratios;

[0262] Iteratively executing the step of obtaining intermediate optical flow detection information and intermediate depth detection information for the upsampled image and the downsampled image corresponding to the target sampling magnification in the image set corresponding to the image frame;

[0263] During any iteration, when the size of the obtained up-sampled image is the same as the size of the image frame, the optical flow detection information and the depth detection information obtained based on the up-sampled image and the image frame are output.

[0264] In one possible implementation, the downsampling module 1002 is configured to:

[0265] Inputting each of the multiple image frames into a plurality of convolutional layers connected in series, performing convolution processing on the image frame through the plurality of convolutional layers to obtain a plurality of downsampled images, wherein each downsampled image is a feature map output by a convolutional layer;

[0266] The image frame and the multiple down-sampled images are determined as an image set corresponding to the image frame.

[0267] In a possible implementation, the target camera is a vehicle-mounted camera, and the multiple image frames are multiple consecutive video frames in a video stream recorded by the vehicle-mounted camera.

[0268] All the above optional technical solutions can be combined in any way to form optional embodiments of the present disclosure, and will not be described in detail here.

[0269] It should be noted that the image processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the processing of image frames. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment are based on the same concept. The specific implementation process is detailed in the image processing method embodiment and will not be repeated here.

[0270] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 11As shown, terminal 1100 is an exemplary embodiment of an electronic device. Optionally, terminal 1100 may be a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. Terminal 1100 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0271] Typically, the terminal 1100 includes a processor 1101 and a memory 1102 .

[0272] Optionally, the processor 1101 includes one or more processing cores, such as a 4-core processor, an 8-core processor, etc. Optionally, the processor 1101 is implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). In some embodiments, the processor 1101 includes a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1101 is integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 also includes an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0273] In some embodiments, the memory 1102 includes one or more computer-readable storage media, optionally, the computer-readable storage medium is non-transitory. Optionally, the memory 1102 also includes a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one program code, which is executed by the processor 1101 to implement the image processing method provided in each embodiment of the present application.

[0274] In some embodiments, terminal 1100 may optionally include a peripheral device interface 1103 and at least one peripheral device. Processor 1101, memory 1102, and peripheral device interface 1103 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1103 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, a positioning assembly 1108, and a power supply 1109.

[0275] The peripheral device interface 1103 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102, and the peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102, and the peripheral device interface 1103 are implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0276] RF circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. RF circuit 1104 optionally includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. RF circuit 1104 optionally communicates with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, RF circuit 1104 also includes circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0277] Display screen 1105 is used to display a UI (User Interface). Optionally, the UI includes graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch screen display, display screen 1105 is also capable of collecting touch signals on or above the surface of display screen 1105. The touch signals can be input as control signals to processor 1101 for processing. Optionally, display screen 1105 is also used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there is a single display screen 1105, provided on the front panel of terminal 1100; in other embodiments, there are at least two display screens 1105, provided on different surfaces of terminal 1100 or in a foldable design; in still other embodiments, display screen 1105 is a flexible display screen, provided on a curved or foldable surface of terminal 1100. Furthermore, display screen 1105 can optionally be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. Optionally, the display screen 1105 is made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0278] The camera assembly 1106 is used to capture images or videos. Optionally, the camera assembly 1106 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1106 also includes a flash. Optionally, the flash is a monochrome temperature flash, or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which is used for light compensation at different color temperatures.

[0279] In some embodiments, the audio circuit 1107 includes a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1101 for processing, or input them into the radio frequency circuit 1104 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there are multiple microphones, which are respectively arranged at different parts of the terminal 1100. Optionally, the microphone is an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. Optionally, the speaker is a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1107 also includes a headphone jack.

[0280] Positioning component 1108 is used to locate the current geographic location of terminal 1100 to implement navigation or LBS (Location Based Service). Optionally, positioning component 1108 is a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Greninja system, or the European Union's Galileo system.

[0281] Power supply 1109 is used to power various components in terminal 1100. Optionally, power supply 1109 is AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 1109 includes a rechargeable battery, the rechargeable battery supports wired charging or wireless charging. The rechargeable battery is also configured to support fast charging technology.

[0282] In some embodiments, the terminal 1100 further includes one or more sensors 1110 , including but not limited to: an acceleration sensor 1111 , a gyroscope sensor 1112 , a pressure sensor 1113 , an optical sensor 1114 , and a proximity sensor 1115 .

[0283] In some embodiments, the accelerometer 1111 detects the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 1100. For example, the accelerometer 1111 is used to detect the components of gravity acceleration along the three coordinate axes. Optionally, the processor 1101 controls the display screen 1105 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 1111. The accelerometer 1111 is also used to collect game or user motion data.

[0284] In some embodiments, the gyroscope sensor 1112 detects the orientation and rotation angle of the terminal 1100. The gyroscope sensor 1112 and the accelerometer sensor 1111 collaborate to capture the user's 3D movements of the terminal 1100. The processor 1101 implements the following functions based on the data collected by the gyroscope sensor 1112: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0285] Optionally, pressure sensor 1113 is provided on the side frame of terminal 1100 and / or below display screen 1105. When pressure sensor 1113 is provided on the side frame of terminal 1100, it can detect the user's gripping signal of terminal 1100, and processor 1101 performs left-hand or right-hand recognition or shortcut operations based on the gripping signal collected by pressure sensor 1113. When pressure sensor 1113 is provided below display screen 1105, processor 1101 controls the operable controls on the UI interface based on the user's pressure operation on display screen 1105. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0286] Optical sensor 1114 is used to detect ambient light intensity. In one embodiment, processor 1101 controls the display brightness of display screen 1105 based on the ambient light intensity detected by optical sensor 1114. Specifically, when the ambient light intensity is high, the display brightness of display screen 1105 is increased; when the ambient light intensity is low, the display brightness of display screen 1105 is decreased. In another embodiment, processor 1101 also dynamically adjusts the capture parameters of camera assembly 1106 based on the ambient light intensity detected by optical sensor 1114.

[0287] Proximity sensor 1115, also known as a distance sensor, is typically located on the front panel of terminal 1100. Proximity sensor 1115 is used to detect the distance between the user and the front of terminal 1100. In one embodiment, when proximity sensor 1115 detects that the distance between the user and the front of terminal 1100 is gradually decreasing, processor 1101 controls display screen 1105 to switch from the screen-on state to the screen-off state. When proximity sensor 1115 detects that the distance between the user and the front of terminal 1100 is gradually increasing, processor 1101 controls display screen 1105 to switch from the screen-off state to the screen-on state.

[0288] Those skilled in the art will understand that Figure 11 The structure shown in the figure does not constitute a limitation on the terminal 1100, and the terminal 1100 can include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0289] Figure 121 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1200 may have relatively large differences due to different configurations or performances. The electronic device 1200 includes one or more processors (Central Processing Units, CPU) 1201 and one or more memories 1202, wherein the memory 1202 stores at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1201 to implement the image processing methods provided in the above-mentioned embodiments. Optionally, the electronic device 1200 also has components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 1200 also includes other components for realizing device functions, which are not described here.

[0290] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program. The at least one computer program can be executed by a processor in a terminal to implement the image processing methods described in each of the above embodiments. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device.

[0291] In an exemplary embodiment, a computer program product or computer program is also provided, including one or more program codes stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more program codes from the computer-readable storage medium and execute the one or more program codes, thereby enabling the electronic device to perform the image processing method of the above-described embodiment.

[0292] Those skilled in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. Optionally, the program is stored in a computer-readable storage medium. Optionally, the above-mentioned storage medium is a read-only memory, a disk or an optical disk, etc.

[0293] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire multiple image frames captured by a target camera, wherein the multiple image frames have a temporal correlation relationship; Downsampling the multiple image frames respectively to obtain a corresponding multiple image sets, each of the multiple image sets including a corresponding image frame and a plurality of downsampled images obtained by downsampling the image frame at different sampling multiples; For any image frame among the multiple image frames and any sampling magnification among the multiple sampling magnifications, obtaining a first down-sampled image corresponding to the sampling magnification from an image set corresponding to the image frame; and obtaining a second down-sampled image corresponding to the sampling magnification from an image set corresponding to a previous frame of the image frame; Obtaining a correlation matrix based on the first down-sampled image and the second down-sampled image, wherein the correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous frame of the image frame at the sampling magnification; Based on the correlation matrix, obtaining intermediate optical flow detection information and intermediate depth detection information; Acquire a third down-sampled image based on the intermediate optical flow detection information, the intermediate depth detection information, and the second down-sampled image, where the third down-sampled image is a down-sampled image corresponding to the sampling magnification obtained by prediction based on a previous frame of the image frame; Upsampling the third downsampled image to obtain an upsampled image corresponding to a target sampling ratio, wherein the target sampling ratio is smaller than and closest to the sampling ratio among the multiple sampling ratios; Iteratively performing the step of obtaining intermediate optical flow detection information and intermediate depth detection information on the upsampled image and the downsampled image corresponding to the target sampling ratio in the image set corresponding to the image frame; During any iteration, when the size of the obtained up-sampled image is the same as the size of the image frame, the optical flow detection information and the depth detection information obtained based on the up-sampled image and the image frame are output.

2. The method according to claim 1, characterized in that The target camera is a binocular camera, the image frame includes a left-eye image and a right-eye image, the first down-sampled image includes a first left-eye down-sampled image and a first right-eye down-sampled image, and the second down-sampled image includes a second left-eye down-sampled image and a second right-eye down-sampled image; The acquiring a correlation matrix based on the first down-sampled image and the second down-sampled image includes: Obtaining a first correlation matrix based on the first left-eye downsampled image and the second left-eye downsampled image, wherein the first correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous left-eye image at the sampling magnification; Obtaining a second correlation matrix based on the first left-eye downsampled image and the first right-eye downsampled image, wherein the second correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of the image frame at the sampling magnification; Obtaining a third correlation matrix based on the second left-eye downsampled image and the second right-eye downsampled image, wherein the third correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of a previous frame of the image frame at the sampling magnification; A fourth correlation matrix is acquired based on the second correlation matrix and the third correlation matrix, where the fourth correlation matrix is used to represent a degree of parallax correlation between the image frame and a previous frame of the image frame at the sampling magnification.

3. The method according to claim 1, characterized in that The acquiring a correlation matrix based on the first down-sampled image and the second down-sampled image includes: For each pixel in the first down-sampled image, determining a sliding window that includes the pixel and a plurality of neighboring pixels of the pixel; Based on the target step size, sliding sampling is performed on the second down-sampled image according to the sliding window to obtain a correlation coefficient between the pixel and each pixel in the second down-sampled image.

4. The method according to claim 3, characterized in that The performing sliding sampling on the second down-sampled image according to the sliding window to obtain a correlation coefficient between the pixel and each pixel in the second down-sampled image includes: Determine, from the first down-sampled image, a first vector consisting of the pixel and a plurality of neighboring pixels of the pixel; For any sliding sampling, determining, from the second down-sampled image, a second vector consisting of a target pixel contained in the sliding window and a plurality of neighboring pixels of the target pixel; A dot product between the first vector and the second vector is determined as a correlation coefficient between the pixel and the target pixel.

5. The method according to claim 1, wherein The downsampling the plurality of image frames respectively to obtain corresponding plurality of image sets comprises: For each of the multiple image frames, input the image frame into a plurality of convolutional layers connected in series, and perform convolution processing on the image frame through the plurality of convolutional layers to obtain a plurality of downsampled images, wherein each downsampled image is a feature map output by a convolutional layer; The image frame and the multiple down-sampled images are determined as an image set corresponding to the image frame.

6. The method according to claim 1, characterized in that The target camera is a vehicle-mounted camera, and the multiple image frames are multiple continuous video frames in a video stream recorded by the vehicle-mounted camera.

7. An image processing device, characterized in that: The device comprises: A first acquisition module is used to acquire multiple image frames captured by a target camera, wherein the multiple image frames have a temporal correlation relationship; a downsampling module, configured to downsample the plurality of image frames respectively to obtain a plurality of corresponding image sets, wherein each image set in the plurality of image sets includes a corresponding image frame and a plurality of downsampled images obtained by downsampling the image frame at different sampling multiples; The second acquisition module includes a first acquisition submodule and a second acquisition submodule, wherein the first acquisition submodule includes a first acquisition unit, a second acquisition unit and a third acquisition unit. The first acquisition unit is configured to acquire, for any image frame among the multiple image frames and any sampling magnification among the multiple sampling magnifications, a first down-sampled image corresponding to the sampling magnification from an image set corresponding to the image frame; and acquire a second down-sampled image corresponding to the sampling magnification from an image set corresponding to a previous frame of the image frame; The second acquisition unit is configured to acquire a correlation matrix based on the first down-sampled image and the second down-sampled image, wherein the correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous frame of the image frame at the sampling magnification; The third acquisition unit is configured to acquire intermediate optical flow detection information and intermediate depth detection information based on the correlation matrix; The second acquisition submodule is used to obtain a third down-sampled image based on the intermediate optical flow detection information, the intermediate depth detection information and the second down-sampled image, where the third down-sampled image is a down-sampled image corresponding to the sampling rate predicted based on the previous frame of the image frame; up-sample the third down-sampled image to obtain an up-sampled image corresponding to a target sampling rate, where the target sampling rate is smaller than and closest to the sampling rate among the multiple sampling rates; iteratively perform the steps of obtaining the intermediate optical flow detection information and the intermediate depth detection information for the up-sampled image and the down-sampled image corresponding to the target sampling rate in the image set corresponding to the image frame; during any iteration, when the size of the obtained up-sampled image is the same as the size of the image frame, output the optical flow detection information and depth detection information obtained based on the up-sampled image and the image frame, where the optical flow detection information is used to characterize the motion of the same object between the image frame and the previous frame, and the depth detection information is used to characterize the distance between the object in the image frame and the target camera.

8. The device according to claim 7, characterized in that The target camera is a binocular camera, the image frame includes a left-eye image and a right-eye image, the first down-sampled image includes a first left-eye down-sampled image and a first right-eye down-sampled image, and the second down-sampled image includes a second left-eye down-sampled image and a second right-eye down-sampled image; The second acquiring unit is configured to: Obtaining a first correlation matrix based on the first left-eye downsampled image and the second left-eye downsampled image, wherein the first correlation matrix is used to represent a degree of pixel correlation between the image frame and a previous left-eye image at the sampling magnification; Obtaining a second correlation matrix based on the first left-eye downsampled image and the first right-eye downsampled image, wherein the second correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of the image frame at the sampling magnification; Obtaining a third correlation matrix based on the second left-eye downsampled image and the second right-eye downsampled image, wherein the third correlation matrix is used to represent the disparity between the left-eye image and the right-eye image of a previous frame of the image frame at the sampling magnification; A fourth correlation matrix is acquired based on the second correlation matrix and the third correlation matrix, where the fourth correlation matrix is used to represent a degree of parallax correlation between the image frame and a previous frame of the image frame at the sampling magnification.

9. The device according to claim 7, characterized in that The second acquiring unit includes: a determining subunit, configured to determine, for each pixel in the first down-sampled image, a sliding window including the pixel and a plurality of neighboring pixels of the pixel; A sampling subunit is configured to perform sliding sampling on the second downsampled image according to the sliding window based on a target step size, so as to obtain a correlation coefficient between the pixel and each pixel in the second downsampled image.

10. The device according to claim 9, characterized in that The sampling subunit is used for: Determine, from the first down-sampled image, a first vector consisting of the pixel and a plurality of neighboring pixels of the pixel; For any sliding sampling, determining, from the second down-sampled image, a second vector consisting of a target pixel contained in the sliding window and a plurality of neighboring pixels of the target pixel; A dot product between the first vector and the second vector is determined as a correlation coefficient between the pixel and the target pixel.

11. The device according to claim 7, characterized in that The downsampling module is used to: For each of the multiple image frames, input the image frame into a plurality of convolutional layers connected in series, and perform convolution processing on the image frame through the plurality of convolutional layers to obtain a plurality of downsampled images, wherein each downsampled image is a feature map output by a convolutional layer; The image frame and the multiple down-sampled images are determined as an image set corresponding to the image frame.

12. The device according to claim 8, characterized in that The target camera is a vehicle-mounted camera, and the multiple image frames are multiple continuous video frames in a video stream recorded by the vehicle-mounted camera.

13. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the image processing method according to any one of claims 1 to 6.

14. A storage medium, characterized in that The storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product includes at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 6.

Images