Parallax matching method and device, robot and storage medium

Abstract

The application discloses a disparity matching method, device, robot and storage medium. The method comprises the following steps: performing epipolar constraint processing on a first feature map and a second feature map to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map, and a matching degree corresponding to each of the plurality of candidate points; the candidate point is a pixel point on a symmetric epipolar line of the to-be-matched point; a target point is selected from the plurality of candidate points according to the matching degree; a matching disparity value output by a disparity matching model is obtained according to the target point, the matching disparity value comprises a coarse disparity value and a correction disparity value corresponding to the to-be-matched point; and a disparity matching point in the second feature map matched with the to-be-matched point in the first feature map is determined according to the coarse disparity value and the correction disparity value. By adopting the method, the accuracy of disparity matching can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computer vision, and in particular to the field of image processing, and more particularly to a disparity matching method and device, robot and storage medium. BACKGROUND

[0002] Binocular stereo vision matching is used to determine the disparity matching points of the same object in binocular images (e.g., left and right eye images taken by a binocular camera).

[0003] In the prior art, for a pixel point in a left eye image, a loss volume is constructed based on all pixel points in a right eye image, i.e., a plurality of similarity values between the left eye pixel point and all right eye pixel points are calculated. Then, a soft argmax function is used to weight and sum the plurality of similarity values to obtain the disparity matching point of the left eye pixel point in the right eye image.

[0004] However, for a pixel point in a left eye image, a disparity matching point can be obtained based on all pixel points in a right eye image, and the accuracy of the disparity matching point is low. SUMMARY

[0005] Therefore, it is necessary to provide a disparity matching method, device and equipment to effectively improve the accuracy of disparity matching.

[0006] In a first aspect, a disparity matching method is provided, comprising:

[0007] obtaining an image pair captured by a binocular camera, the image pair comprising a left eye image and a right eye image;

[0008] inputting the left eye image and the right eye image into a disparity matching model to respectively perform feature extraction and obtain a first feature map and a second feature map;

[0009] performing epipolar constraint processing on the first feature map and the second feature map to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map, and a matching degree corresponding to each of the plurality of candidate points; the candidate point is a pixel point on a symmetric epipolar line of the to-be-matched point;

[0010] selecting a target point from the plurality of candidate points according to the matching degree;

[0011] obtaining a matching disparity value output by the disparity matching model according to the target point, the matching disparity value comprising a coarse disparity value and a corrected disparity value corresponding to the to-be-matched point;

[0012] determining a disparity matching point in the second feature map that matches the to-be-matched point in the first feature map according to the coarse disparity value and the corrected disparity value.

[0013] In a second aspect, a disparity matching device is provided, and the device comprises:

[0014] An acquisition unit is configured to acquire an image pair collected by a binocular camera, the image pair comprising a left-eye image and a right-eye image;

[0015] A feature extraction unit is configured to input the left-eye image and the right-eye image into a disparity matching model to perform feature extraction on the left-eye image and the right-eye image respectively, and obtain a first feature map and a second feature map;

[0016] A processing unit is configured to perform epipolar constraint processing on the first feature map and the second feature map to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map, and a matching degree corresponding to each of the plurality of candidate points; the candidate point is a pixel point on a symmetric epipolar line of the to-be-matched point;

[0017] A target point screening unit is configured to screen a target point from the plurality of candidate points according to the matching degree;

[0018] A matching disparity value output unit is configured to obtain a matching disparity value output by the disparity matching model according to the target point, the matching disparity value comprising a coarse disparity value and a corrected disparity value corresponding to the to-be-matched point;

[0019] A disparity matching point determination unit is configured to determine a disparity matching point in the second feature map matched with the to-be-matched point in the first feature map according to the coarse disparity value and the corrected disparity value.

[0020] In a third aspect, a robot is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method of the first aspect and any possible implementation manner of the first aspect are implemented.

[0021] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method of the first aspect and any possible implementation manner of the first aspect are implemented.

[0022] In the method of the present application, an image pair collected by a binocular camera is obtained, the image pair comprising a left-eye image and a right-eye image; the left-eye image and the right-eye image are input into a disparity matching model for feature extraction to obtain a first feature map and a second feature map; the first feature map and the second feature map are subjected to epipolar constraint processing to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map and a matching degree corresponding to each of the plurality of candidate points; the candidate point is a pixel point on a symmetric epipolar line of the to-be-matched point; a target point is selected from the plurality of candidate points according to the matching degree; a matching disparity value output by the disparity matching model is obtained according to the target point, the matching disparity value comprising a coarse disparity value and a corrected disparity value corresponding to the to-be-matched point; and a disparity matching point in the second feature map matched with the to-be-matched point in the first feature map is determined according to the coarse disparity value and the corrected disparity value.

[0023] Compared with the prior art, in the method of the present application, the epipolar constraint rule is used to reduce the range of the candidate point from a plane to a straight line, and the disparity matching point is determined based on the coarse disparity value and the corrected disparity value, thereby effectively improving the accuracy of disparity matching. BRIEF DESCRIPTION OF DRAWINGS

[0024] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0025] Figure 1 A hardware structure schematic diagram of a robot provided for an embodiment of the present application;

[0026] Figure 2 A mechanical structure schematic diagram of a robot provided for an embodiment of the present application;

[0027] Figure 3 A system block diagram of a disparity matching system provided for an embodiment of the present application;

[0028] Figure 4 A flowchart of a disparity matching method provided for an embodiment of the present application;

[0029] Figure 5 A principle diagram of epipolar constraint provided for an embodiment of the present application;

[0030] Figure 6 Another flowchart of a disparity matching method provided for an embodiment of the present application;

[0031] Figure 7 Another flowchart of a disparity matching method provided for an embodiment of the present application;

[0032] Figure 8 Another flowchart of a disparity matching method provided for an embodiment of the present application;

[0033] Figure 9 a training process of the parallax matching model provided by the embodiment of the present application;

[0034] Figure 10 a flow chart of the training process of the parallax matching model provided by the embodiment of the present application;

[0035] Figure 11 a structure schematic diagram of the parallax matching device provided by the embodiment of the present application;

[0036] Figure 12 a structure block diagram of the computer device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described below in conjunction with the embodiments and the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain the related application, but not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for the convenience of description.

[0038] It should be noted that the features of the embodiments in the present application, i.e. the features of the embodiments, can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0039] Please refer to Figure 1 , Figure 1 A hardware structure schematic diagram of the robot 100 is provided by one of the embodiments of the present application. The robot 100 can be any one of various forms of robots, specifically including but not limited to at least one of a wheeled robot, a legged robot, a tracked robot, a crawling robot, an inchworm robot, or a swimming robot, etc. For example, the robot 100 can be specifically a legged robot, or a robot combining a legged robot and a wheeled robot. The legged robot includes a single-legged robot, a biped robot, or a multi-legged robot. The multi-legged robot refers to a legged robot with three or more legs, for example, the multi-legged robot can be specifically a quadruped robot. The robot refers to a machine capable of semi-autonomous or fully autonomous execution of work, and the robot is not limited to a humanoid machine device, but can also include a robot with a configuration such as a dog shape, a horse shape, a snake shape, a fish shape, etc. For example, the robot can be specifically a four-legged mechanical horse. In the embodiment shown in the figure, the robot 100 includes a mechanical unit 101, a communication unit 102, a sensing unit 103, an interface unit 104, a storage unit 105, a display unit 106, an input unit 107, a control module 110, and a power supply 111. The various components of the robot 100 can be connected in any way, including wired or wireless connection, etc. Those skilled in the art can understand that the robot 100 can include more components, or some components can be omitted, and the components can be connected in other ways. Figure 1 In the embodiment shown, the robot 100 includes a mechanical unit 101, a communication unit 102, a sensing unit 103, an interface unit 104, a storage unit 105, a display unit 106, an input unit 107, a control module 110, and a power supply 111. The various components of the robot 100 can be connected in any way, including wired or wireless connection, etc. Those skilled in the art can understand that the robot 100 can include more components, or some components can be omitted, and the components can be connected in other ways. Figure 1The specific structure of the robot 100 shown in the figure does not constitute a limitation on the robot 100, and the robot 100 can include more or fewer components than shown in the figure, and some components are not necessarily included in the robot 100, and some components can be omitted as needed without changing the essence of the application, or some components can be combined.

[0040] Figure 2 A mechanical structure schematic diagram of a robot according to an embodiment of the application is provided.

[0041] The specific structure of the robot 100 shown in the figure does not constitute a limitation on the robot 100, and the robot 100 can include more or fewer components than shown in the figure, and some components are not necessarily included in the robot 100, and some components can be omitted as needed without changing the essence of the application, or some components can be combined. Figure 1 Figure 2 The specific structure of the robot 100 shown in the figure does not constitute a limitation on the robot 100, and the robot 100 can include more or fewer components than shown in the figure, and some components are not necessarily included in the robot 100, and some components can be omitted as needed without changing the essence of the application, or some components can be combined.

[0042] The mechanical unit 101 is the hardware of the robot 100. As shown in the figure, Figure 1 the mechanical unit 101 can include a drive board 1011, a motor 1012, and a mechanical structure 1013. As shown in the figure, Figure 2 the mechanical structure 1013 can include a body main body 1014, an extendable leg 1015, a foot 1016, and in other embodiments, the mechanical structure 1013 can also include an extendable mechanical arm (not shown in the figure), a rotatable head structure 1017, a swingable tail structure 1018, a load structure 1019, a saddle structure 1020, a camera structure 1021, etc. It should be noted that each component module of the mechanical unit 101 can be one or more, which can be set according to specific circumstances, such as the leg 1015 can be 4, and each leg 1015 can be configured with 3 motors 1012, and the corresponding motors 1012 are 12.

[0043] The communication unit 102 can be used for signal reception and transmission, and can also communicate with the network and other devices, such as receiving the instruction information sent by the remote controller or other robots 100 to move in a specific gait at a specific speed value in a specific direction, and then transmitting it to the control module 110 for processing. The communication unit 102 includes, for example, a WiFi module, a 4G module, a 5G module, a Bluetooth module, an infrared module, etc.

[0044] ​The sensing unit 103 is configured to acquire information data of the environment around the robot 100 and monitor parameter data of components inside the robot 100, and transmit the acquired information data and the monitored parameter data to the control module 110. The sensing unit 103 includes various sensors, such as sensors configured to acquire information of the environment around the robot 100, such as a laser radar (for long-range object detection, distance determination, and / or speed value determination), a millimeter wave radar (for short-range object detection, distance determination, and / or speed value determination), a camera, an infrared camera, a Global Navigation Satellite System (GNSS), and the like. The sensing unit 103 also includes sensors configured to monitor components inside the robot 100, such as an Inertial Measurement Unit (IMU) (for measuring values of speed, acceleration, and angular velocity), a plantar sensor (for monitoring plantar contact point positions, plantar postures, ground contact force magnitudes, and directions), and a temperature sensor (for detecting component temperatures). The robot 100 can also be equipped with other sensors, such as a load sensor, a touch sensor, a motor angle sensor, a torque sensor, and the like, which are not described herein.

[0045] The interface unit 104 can be configured to receive input (e.g., data information, power, and the like) from an external device and transmit the received input to one or more components inside the robot 100, or can be configured to output (e.g., data information, power, and the like) to an external device. The interface unit 104 can include a power supply port, a data port (e.g., a USB port), a memory card port, a port for connecting a device having an identification module, an audio input / output (I / O) port, a video I / O port, and the like.

[0046] The storage unit 105 is configured to store software programs and various data. The storage unit 105 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system program, a motion control program, an application program (such as a text editor), and the like; and the data storage area can store data generated by the robot 100 in use (such as various sensing data acquired by the sensing unit 103, log file data), and the like. In addition, the storage unit 105 can include a high-speed random access memory, and can also include a non-volatile memory, such as a disk memory, a flash memory, or other volatile solid-state memory.

[0047] The display unit 106 is configured to display information input by a user or information provided to a user. The display unit 106 can include a display panel 1061, which can be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

[0048] The input unit 107 can be configured to receive input digital or character information. Specifically, the input unit 107 can include a touch panel 1071 and other input devices 1072. The touch panel 1071, also called a touch screen, can collect touch operations of a user (such as the user's operation on or near the touch panel 1071 using a palm, a finger, or a suitable accessory) and drive the corresponding connection device according to a pre-set program. The touch panel 1071 can include two parts, a touch detection device 1073 and a touch controller 1074. The touch detection device 1073 detects the touch position of the user and detects the signal generated by the touch operation, and transmits the signal to the touch controller 1074; the touch controller 1074 receives the touch information from the touch detection device 1073, converts it into touch coordinates, and sends it to the control module 110, and can also receive the command sent by the control module 110 and execute it. In addition to the touch panel 1071, the input unit 107 can also include other input devices 1072. Specifically, the other input devices 1072 can include one or more of, but are not limited to, a remote control handle, etc., and the specific embodiments are not limited here.

[0049] Further, the touch panel 1071 can cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits to the control module 110 to determine the type of touch event, and then the control module 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Although in the Figure 1 embodiments, the touch panel 1071 and the display panel 1061 are implemented as two independent components to realize input and output functions, respectively, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize input and output functions, and the specific embodiments are not limited here.

[0050] The control module 110 is the control center of the robot 100, which connects all components of the robot 100 through various interfaces and lines, and controls the whole robot 100 by running or executing the software program stored in the storage unit 105 and calling the data stored in the storage unit 105.

[0051] The power supply 111 is used to supply power to each component. The power supply 111 can include a battery and a power control board, which is used to control the functions of battery charging, discharging, and power consumption management, etc. In the embodiment shown in Figure 1 The power supply 111 is electrically connected to the control module 110. In other embodiments, the power supply 111 can also be electrically connected to the sensing unit 103 (such as a camera, a radar, a sound box, etc.) and the motor 1012, respectively. It should be noted that each component can be connected to different power supplies 111 or powered by the same power supply 111.

[0052] On the basis of the above-mentioned embodiments, specifically, in some embodiments, the terminal device can be connected in communication with the robot 100, and when the terminal device communicates with the robot 100, the terminal device can send instruction information to the robot 100, the robot 100 can receive the instruction information through the communication unit 102, and can transmit the instruction information to the control module 110 when receiving the instruction information, so that the control module 110 can process the target speed value according to the instruction information. The terminal device includes but is not limited to: a mobile phone with image shooting function, a tablet computer, a server, a personal computer, a wearable smart device, and other electrical appliances.

[0053] The instruction information can be determined according to a preset condition. In an embodiment, the robot 100 can include a sensing unit 103, which can generate instruction information according to the current environment in which the robot 100 is located. The control module 110 can determine whether the current speed value of the robot 100 meets the corresponding preset condition according to the instruction information. If it meets, the current speed value and the current gait of the robot 100 will be maintained; if it does not meet, the target speed value and the corresponding target gait will be determined according to the corresponding preset condition, so that the robot 100 can be controlled to move at the target speed value and the corresponding target gait. The environmental sensor can include a temperature sensor, a barometric pressure sensor, a visual sensor, and a sound sensor. The instruction information can include temperature information, barometric pressure information, image information, and sound information. The communication mode between the environmental sensor and the control module 110 can be wired communication or wireless communication. The wireless communication mode includes but is not limited to: wireless network, mobile communication network (3G, 4G, 5G, etc.), Bluetooth, and infrared.

[0054] The foregoing introduces the schematic diagram of the hardware structure and the schematic diagram of the mechanical structure of the robot. The following introduces the prior art and the technical problems of the prior art in the process of disparity matching.

[0055] In the prior art, for a pixel point in the left eye image, a loss volume is constructed based on all pixel points in the right eye image, that is, a plurality of similarity values between the left eye pixel point and all right eye pixel points are calculated. Then, a soft argmax function is used to weight and sum the plurality of similarity values to obtain the disparity matching point of the left eye pixel point in the right eye image.

[0056] However, for a pixel point in the left eye image, a disparity matching point can be obtained based on all pixel points in the right eye image, and the accuracy of the disparity matching point is low.

[0057] Therefore, the present application proposes a disparity matching method, device, robot and storage medium, which can effectively improve the accuracy of disparity matching.

[0058] The disparity matching method provided in the present application can be applied to a disparity matching system as shown in Figure 3 The system can include a robot 10 and a background server 20. Figure 3 The system can include a robot 10 and a background server 20.

[0059] The robot 10 can obtain binocular image (e.g., left eye and right eye) information of the surrounding road, and generate a disparity map according to the binocular image information, and then determine the depth information of the robot from a certain object according to the disparity map as auxiliary information for the robot to travel.

[0060] The background server 20 supports the background implementation of various functions provided by the robot 10. For example, the background server 20 can receive instructions sent by the robot 10, so that the background server 20 executes and returns the execution result of the instructions to the robot 10.

[0061] The above-mentioned background server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0062] In another embodiment of the present application, a disparity matching method is provided, which can be applied to the robot 10 or the background server 20 in Figure 3 As shown in Figure 4 The method includes the following steps:

[0063] Step 401, obtaining an image pair collected by a binocular camera, the image pair including a left eye image and a right eye image.

[0064] It can be understood that during the robot travels, the robot can obtain binocular images of the surrounding environment, generate a disparity map based on the binocular images, and determine the depth information of the robot from a certain object according to the disparity map, thereby providing protection for the safe travel of the robot.

[0065] Among them, the image pair is a pair of images collected by a binocular camera obtained by the robot, and the pair of images includes two types of images. One is a left eye image, that is, an image captured by a left eye camera of the binocular camera for the same shooting object (e.g., the same object or the same picture), and the other is a right eye image, that is, an image captured by a right eye camera of the binocular camera for the same shooting object.

[0066] In a possible implementation, the robot can acquire or collect an image pair of the binocular camera for a same shooting object (for example, a same object or a same picture), that is, a left-eye image and a right-eye image of the same shooting object.

[0067] In step 402, the left-eye image and the right-eye image are input into a disparity matching model, and feature extraction is respectively performed to obtain a first feature map and a second feature map.

[0068] The disparity matching model is configured to output a disparity value between a to-be-matched point in the first feature map and a disparity-matched point matched with the to-be-matched point in the second feature map.

[0069] The first feature map can be an image obtained after feature extraction is performed on the left-eye image or the right-eye image.

[0070] The second feature map can be an image obtained after feature extraction is performed on the left-eye image or the right-eye image.

[0071] In a possible implementation, the FPN (Feature Pyramid Networks) network in the disparity matching model can be used to perform feature extraction on the left-eye image and the right-eye image to obtain a feature map of the left-eye image and a feature map of the right-eye image.

[0072] It should be noted that, if the feature map of the left-eye image is taken as the first feature map, it can be understood that the first feature map is taken as a reference feature map, and a disparity-matched point matched with a to-be-matched point in the reference feature map is determined in the second feature map (that is, the feature map of the right-eye image).

[0073] If the feature map of the right-eye image is taken as the first feature map, it can be understood that the first feature map is taken as a reference feature map, and a disparity-matched point matched with a to-be-matched point in the reference feature map is determined in the second feature map (that is, the feature map of the left-eye image).

[0074] In step 403, epipolar constraint processing is performed on the first feature map and the second feature map to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map, and a matching degree corresponding to each of the plurality of candidate points; the candidate point is a pixel point of the to-be-matched point on a symmetric epipolar line.

[0075] The candidate point is a pixel point of the to-be-matched point on a symmetric epipolar line.

[0076] The concept of the symmetric epipolar line is described below. Figure 5 A principle diagram of the epipolar constraint provided by the embodiment of the present application is shown in FIG. 1. Figure 5 Point P is a shooting object (equivalent to the same shooting object in the foregoing embodiment), point O lO is the optical center of the left camera of the binocular camera r P is the optical center of the right camera of the binocular camera, and plane L is the plane passing through point O and point P l After projection, the plane on which the projection point is located is the left projection plane. Plane R is the plane passing through point O and point P r After projection, the plane on which the projection point is located is the right projection plane. PO l O r The intersection line of the plane (i.e., the epipolar plane) composed of points O and P and the plane L is the left epipolar line, and the intersection line of the plane composed of points O and P and the plane R is the right epipolar line. If the projection point p of point P on the plane L falls on the left epipolar line, then the projection point p' of point P on the plane R, or the parallax matching point p' of point p on the plane R, must fall on the right epipolar line.

[0077] The matching degree is explained by taking one to-be-matched point as an example. The matching degree is the similarity degree or the difference degree between the to-be-matched point and multiple candidate points. For example, the similarity degree can be used to represent the similarity degree, or the difference degree can be used to represent the difference degree. It can be understood that the to-be-matched point has a higher similarity degree with a certain candidate point, which can be equivalent to that the to-be-matched point has a lower difference degree with the certain candidate point. Or, the to-be-matched point has a lower similarity degree with a certain candidate point, which can be equivalent to that the pixel point has a higher difference degree with the certain candidate point.

[0078] In a possible implementation, the epipolar line constraint processing can be performed on the first feature map and the second feature map according to the epipolar line constraint rule.

[0079] In a possible implementation, after the epipolar line constraint processing is performed on the first feature map and the second feature map, multiple candidate points p1-p4 of the to-be-matched point in the second feature map can be obtained, and the matching degrees of the multiple candidate points, such as the similarity degrees s1-s4 or the difference degrees d1-d4, can be obtained.

[0080] Step 404: screening a target point from the multiple candidate points according to the matching degrees.

[0081] The target point is a point with the highest similarity degree or the lowest difference degree with the to-be-matched point in the multiple candidate points. For example, the target point can be point p3.

[0082] Step 405: obtaining a matching parallax value output by the parallax matching model according to the target point, and the matching parallax value includes a coarse parallax value and a corrected parallax value corresponding to the to-be-matched point.

[0083] First, the concept of the coarse disparity value is explained. If the projection point of the photographed object point P in the left eye projection plane L is point p, the coordinate of point p can be (x1, y1). Since the binocular camera is similar to the human eyes, for the same photographed object point P, a pair of matching projection points will have disparity, i.e., the coordinates of point p and point p' will be different. For example, if the projection point of the photographed object point P in the right eye projection plane is point p', the coordinate of point p' can be (x3, y1). In the right eye projection plane, there is a point (x1, y1) which is the same as point p, and the integer part of the pixel difference value between x1 and x3 is the coarse disparity value. If the first feature map is taken as the reference feature map, the coarse disparity value can be the pixel difference value in the horizontal direction between the target point p3 and point p in the second feature map. For example, the pixel difference value 3 in the horizontal direction between p3 (x1+3, y1) and point p (x1, y1) is the integer part of the pixel difference value between x1 and x3.

[0084] Second, the concept of the modified disparity value is explained. If the projection point of the photographed object point P in the left eye projection plane L is point p, the coordinate of point p can be (x1, y1). Since the binocular camera is similar to the human eyes, for the same photographed object point P, a pair of matching projection points will have disparity, i.e., the coordinates of point p and point p' will be different. For example, if the projection point of the photographed object point P in the right eye projection plane is point p', the coordinate of point p' can be (x3, y1). In the right eye projection plane, there is a point (x1, y1) which is the same as point p, and the decimal part of the pixel difference value between x1 and x3 is the modified disparity value. The modified disparity value is a sub-pixel level disparity value. The modified disparity value can be a pixel value determined according to the interval corresponding to the field pixel point of the point (e.g., point p3) with the highest matching degree in the candidate points.

[0085] In a possible implementation, the point p3 corresponding to the maximum value s3 of the similarity s1-s4 can be taken as the target point based on the argmax function according to the matching degrees of the candidate points, e.g., according to the similarity s1-s4, or the point p3 corresponding to the minimum value d3 of the difference degree d1-d4 can be taken as the target point based on the argmin function according to the difference degree d1-d4.

[0086] In a possible implementation, the pixel value 3 of point p3 can be determined as the coarse disparity value.

[0087] In a possible implementation, if the target point is p3, the pixel value of point p3 can be determined according to the function The pixel interval (i.e., the pixel interval in the horizontal direction) of the point p2 and the point p4 is mapped to (-0.5, 0.5), for example, if the horizontal pixel value x2 of the point p2 is 2 and the horizontal pixel value x4 of the point p4 is 4, the pixel interval can be (2, 4), and the mapping result can be 0.2. Alternatively, any pixel value in any pixel interval can be mapped to (-0.5, 0.5) according to other functions. That is, the mapping region of the pixel value is not limited in the embodiments of the present application, and the mapping result can be in (-0.5, 0.5).

[0088] In a possible implementation, the matching disparity value 3.2 output by the disparity matching model can include the coarse disparity value 3 and the correction disparity value 0.2 corresponding to the to-be-matched point.

[0089] In step 406, the disparity matching point in the second feature map matched with the to-be-matched point in the first feature map is determined according to the coarse disparity value and the correction disparity value.

[0090] The disparity matching point is explained by taking one to-be-matched point as an example. The disparity matching point is the point in the second feature map matched with the to-be-matched point in the first feature map. For example, the disparity matching point matched with the to-be-matched point p can be the point p'.

[0091] In a possible implementation, taking the determination of the disparity matching point of one to-be-matched point as an example, the coarse disparity value 3 and the correction disparity value 0.2 can be superimposed to obtain the matching disparity value 3.2, and then the point p' matched with the point p having the same coordinates in the second feature map is determined according to the matching disparity value 3.2. For example, if the point p is (x1, y1), the matching disparity value 3.2 can be superimposed with x1 to obtain 3.2+x1, and then the disparity matching point p' is (3.2+x1, y1).

[0092] In the prior art, for one pixel point in the left eye image, a loss volume is constructed based on all pixel points in the right eye image, that is, a plurality of similarity values between the left eye pixel point and all right eye pixel points are calculated. Then, a soft argmax function is used to perform weighted summation on the plurality of similarity values to obtain the disparity matching point of the left eye pixel point in the right eye image.

[0093] In the method of the present application, an image pair collected by a binocular camera is obtained, the image pair comprising a left-eye image and a right-eye image; the left-eye image and the right-eye image are input into a disparity matching model, and feature extraction is performed respectively to obtain a first feature map and a second feature map; epipolar constraint processing is performed on the first feature map and the second feature map to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map, and a matching degree corresponding to each of the plurality of candidate points; the candidate point is a pixel point on a symmetric epipolar line of the to-be-matched point; a target point is selected from the plurality of candidate points according to the matching degree; a matching disparity value output by the disparity matching model is obtained according to the target point, the matching disparity value comprising a coarse disparity value and a corrected disparity value corresponding to the to-be-matched point; and a disparity matching point in the second feature map matched with the to-be-matched point in the first feature map is determined according to the coarse disparity value and the corrected disparity value.

[0094] Compared with the prior art, first, the accuracy of disparity matching is improved due to the adoption of the disparity matching model. Second, the range of the candidate point is reduced from a plane to a straight line by adopting the epipolar constraint rule, thereby improving the accuracy of disparity matching. Third, the coarse disparity value is corrected by the corrected disparity value, thereby further improving the accuracy of disparity matching and making the obtained disparity matching point more accurate.

[0095] In the foregoing embodiment, the determination of the disparity matching point is introduced. In another embodiment of the present application, how to determine the plurality of candidate points is introduced. For example, the specific implementation of the step of "performing epipolar constraint processing on the first feature map and the second feature map to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map" in the foregoing step comprises the following steps: Figure 6

[0096] Step 601: Taking a to-be-matched point in the first feature map as a search reference, taking a preset offset as a search step, and obtaining a plurality of similar pixel points in the second feature map on a symmetric epipolar line of the to-be-matched point.

[0097] The search reference is explained by taking one to-be-matched point as an example. The search reference can be a search center taking the to-be-matched point in the first feature map as the search center. For example, the search center can be point p(x1, y1).

[0098] The preset offset can be a search range. For example, the search range can be a circle with a fixed radius. For example, it can be a circle with a radius of r1.

[0099] The search step can be a search range. The search range can be circular or square.

[0100] ​The similar pixel point can be a pixel point in the second feature map that is on a symmetric polar line with the to-be-matched point in the first feature map. For example, the coordinates of the to-be-matched point can be point (x1, y1), and the similar pixel points can be (x1+1, y1), (x1+2, y1), (x1+3, y1), (x1+4, y1), (x1+5, y2), (x1+6, y2).

[0101] In a possible implementation, the to-be-matched point p (x1, y1) in the first feature map can be taken as a search reference, and a preset offset r1 can be taken as a search step size, to obtain a plurality of similar pixel points p1 (x1+1, y1), p2 (x1+2, y1), p3 (x1+3, y1), p4 (x1+4, y1), p5 (x1+5, y2), and p6 (x1+6, y2) in the second feature map that are on a symmetric polar line with the to-be-matched point p (x1, y1).

[0102] In step 602, a mapping point with the same position as the to-be-matched point is determined in the second feature map, and a plurality of similar pixel points adjacent to the mapping point are taken as candidate points.

[0103] The mapping point can be a point in the second feature map that has the same position as the to-be-matched point. For example, if the to-be-matched point in the first feature map is p (x1, y1), the mapping point can be a point (x1, y1) in the second feature map that has the same position as the to-be-matched point p.

[0104] In a possible implementation, a plurality of similar pixel points adjacent to the to-be-matched point p (x1, y1) can be selected from the plurality of similar pixel points, i.e., point p1 (x1+1, y1), point p2 (x1+2, y1), point p3 (x1+3, y1), point p4 (x1+4, y1), point p5 (x1+5, y2), and point p6 (x1+6, y2), as candidate points. For example, p1 (x1+1, y1), point p2 (x1+2, y1), point p3 (x1+3, y1), and point p4 (x1+4, y1) can be taken as candidate points.

[0105] It should be noted that adjacent is used to represent that the distance between the mapping point and the similar pixel point is within a preset range. For example, the preset range can be a range in which the difference between the horizontal coordinates of the mapping point and the similar pixel point is not more than 4, and the similar pixel points in the range can be taken as candidate points corresponding to the to-be-matched point. For example, p1 (x1+1, y1), point p2 (x1+2, y1), point p3 (x1+3, y1), and point p4 (x1+4, y1) can be taken as candidate points.

[0106] It can be seen that the similar pixel points near the mapping point are taken as the candidate points, the range of the candidate points is reduced from the epipolar line to several points, the accuracy of the candidate points is improved, and further, the accuracy of determining the disparity matching point based on the candidate points is improved.

[0107] In the foregoing embodiment, how to determine the plurality of candidate points is introduced. In another embodiment of the present application, how to determine the coarse disparity value in the matching disparity value is introduced. For example, the specific implementation of the step of "obtaining the matching disparity value output by the disparity matching model according to the target point" involved in the foregoing embodiment includes the steps of: Figure 7

[0108] Step 701, determining a mapping point same as the position of the to-be-matched point in the second feature map.

[0109] The mapping point can be a point same as the position of the to-be-matched point in the second feature map. For example, if the to-be-matched point in the first feature map is p(x1, y1), the mapping point can be a point (x1, y1) same as the position of the to-be-matched point p in the second feature map.

[0110] In a possible implementation, if the to-be-matched point in the first feature map is p(x1, y1), the point (x1, y1) in the second feature map can be taken as the mapping point.

[0111] Step 702, taking the pixel difference value between the mapping point and the target point as the coarse disparity value.

[0112] In a possible implementation, the pixel difference value 3 between the pixel value (x1+3, y1) of the target point p3 and the to-be-matched point p(x1, y1) can be taken as the coarse disparity value. It can be understood that since the y coordinates of the point p3 and the point p are the same, the coarse disparity value can be 3.

[0113] It can be seen that the disparity matching point can be determined based on the coarse disparity value, and the accuracy of the disparity matching point is improved.

[0114] In the foregoing embodiment, how to determine the coarse disparity value in the matching disparity value is introduced. In another embodiment of the present application, how to determine the corrected disparity value in the matching disparity value is introduced. For example, the specific implementation of the step of "obtaining the matching disparity value output by the disparity matching model according to the target point" involved in the foregoing embodiment includes the steps of: Figure 8

[0115] Step 801, determining a corrected disparity value corresponding to the to-be-matched point according to the neighborhood point corresponding to the target point.

[0116] The neighborhood point can be a point adjacent to the target point. For example, if the target point is p3, the point adjacent to the target point can be p2 or p4.​​

[0117] In a possible implementation, if the target point is p3, any pixel value in the pixel interval (x1+2, x1+4) of the point p2 and the point p4 can be mapped to (-0.5, 0.5) according to a sigmoid function. It should be noted that the modified disparity value in the embodiment of the present application is the difference of the horizontal coordinates in the two-dimensional space. For example, the difference of the x coordinates.

[0118] In step 802, the pixel values in the neighborhood point interval are mapped to obtain a modified disparity value in a preset numerical interval; the preset numerical interval is a decimal interval.

[0119] The neighborhood point interval can be an interval formed by the target point and the adjacent point. For example, if the target point is p3, the interval formed by the target point and the adjacent point can be the interval (x1+2, x1+4) formed by p2 and p4. It should be noted that the modified disparity value in the embodiment of the present application is the difference of the horizontal coordinates in the two-dimensional space. For example, the difference of the x coordinates.

[0120] The preset interval can be a decimal interval with specific upper and lower limits. For example, it can be (-0.5, 0.5).

[0121] In a possible implementation, if p2 is (x1+2, y1) and p4 is (x1+4, y1), the pixel interval can be (x1+2, x1+4). The sigmoid function can be used to map any value in the interval (2, 4) to obtain the modified disparity value 0.2 in the preset interval (-0.5, 0.5). Alternatively, any pixel value in any pixel interval can be mapped to (-0.5, 0.5) according to other functions. That is, the mapping region of the pixel value is not specifically limited in the embodiment of the present application, as long as the mapping result is in (-0.5, 0.5).

[0122] As can be seen, the disparity matching point can be determined based on the modified disparity value, and the accuracy of the disparity matching point is improved.

[0123] In the foregoing embodiments, how to determine the modified disparity value in the matching disparity value is introduced. In another embodiment of the present application, how to determine the disparity matching point is introduced. For example, the specific implementation of the step in the foregoing embodiment, “determining the disparity matching point in the second feature map matched with the target matching point in the first feature map according to the coarse disparity value and the modified disparity value”, includes:

[0124] The coarse disparity value, the modified disparity value, and the pixel value of the mapping point are superimposed to obtain the disparity matching point of each target matching point in the first feature map in the second feature map.

[0125] The mapping point can be a point in the second feature map that has the same position as the point p (i.e., the to-be-matched point in the first feature map). For example, (x1, y1).

[0126] The pixel value of the mapping point can be x1. It should be noted that the pixel value in the embodiment of the present application is the difference of the horizontal coordinate in the two-dimensional space. For example, the difference of the x coordinate, and the y coordinate are the same.

[0127] In a possible implementation, the coarse disparity value 3 and the corrected disparity value 0.2 can be superimposed to obtain a matching disparity value 3.2, and then 3.2 is added to the pixel value x1 of the mapping point to obtain the disparity matching point p' corresponding to the to-be-matched point p (x1, y1), which is (3.2+x1, y1). It should be noted that the vertical coordinate of the disparity matching point in the embodiment of the present application is the same as that of the to-be-matched point, and in the process of calculating the matching disparity value, only the difference of the horizontal coordinate is considered.

[0128] As can be seen, the coarse disparity value and the corrected disparity matching value are used to obtain the disparity matching point based on the matching disparity value, which improves the accuracy of the disparity matching point.

[0129] In the foregoing embodiment, how to determine the disparity matching point is introduced. In another embodiment of the present application, the training process of the disparity matching model is introduced. For example, the specific implementation of the training process of the disparity matching model includes the following steps: Figure 9

[0130] Step 901, inputting a sample image pair into an initial disparity matching model to perform feature extraction and obtaining a feature map of the sample image pair.

[0131] The sample image pair is a pair of images collected by a binocular camera, and the pair of images includes two types of images. One type is a left-eye image, that is, an image captured by a left-eye camera of the binocular camera for the same photographed object (for example, the same object or the same picture), and the other type is a right-eye image, that is, an image captured by a right-eye camera of the binocular camera for the same photographed object.

[0132] The initial disparity matching model can be an untrained disparity matching model.

[0133] The feature map of the sample image pair can be a feature map of the left-eye image or a feature map of the right-eye image.

[0134] In a possible implementation, an FPN (Feature Pyramid Networks) network can be used to perform feature extraction on the left-eye image and the right-eye image.

[0135] ​In a possible implementation, after the FPN (Feature Pyramid Networks) network performs feature extraction on the left-eye image and the right-eye image, a feature map of the left-eye image and a feature map of the right-eye image can be obtained.

[0136] In step 902, after processing the feature maps of the sample image pair by using the initial disparity model, a coarse disparity value and a corrected disparity value are obtained, and a matching disparity value is determined according to the coarse disparity value and the corrected disparity value.

[0137] The matching disparity value is a disparity value output by the initial disparity model for the feature maps of the sample image pair, and the matching disparity value is a sum of the coarse disparity value and the corrected disparity value.

[0138] In a possible implementation, since the feature maps of the sample image pair include the feature map of the left-eye image and the feature map of the right-eye image, if the feature map of the left-eye image is used as a reference feature map, the initial disparity model can output a coarse disparity value and a corrected disparity value corresponding to the feature map of the right-eye image as 3 and 0.1 respectively.

[0139] In step 903, the initial disparity matching model is iteratively trained based on a loss between the matching disparity value and a label disparity value, and a disparity matching model is obtained.

[0140] For example, the label disparity value is an actual or real disparity value between a to-be-matched point in the feature map of the left-eye image and a disparity matching point corresponding to the to-be-matched point in the feature map of the right-eye image.

[0141] The disparity matching model is a model that can be used to output the matching disparity value after training.

[0142] In a possible implementation, if the initial disparity model predicts that the matching disparity value is 3.1, and the label disparity value is 3.2, there is a loss between 3.1 and 3.2, and the loss can act on a network corresponding to the initial disparity model. The network can adjust parameters so that the loss becomes smaller and smaller. If the matching disparity value is equal to the label disparity value, that is, the loss tends to 0, it indicates that the initial disparity model becomes the disparity model after training.

[0143] It can be seen that after the initial disparity matching model is trained, the disparity matching model can be obtained, the disparity matching model can output a disparity matching value, and the accuracy of the disparity matching point can be improved based on the disparity matching value.

[0144] In the foregoing embodiment, the training process of the disparity matching model is introduced. In another embodiment of the present application, it is introduced that what kind of loss is included in the loss between the matching disparity value and the label disparity value. For example, the loss between the matching disparity value and the label disparity value described in the foregoing embodiment includes a first loss, a second loss.

[0145] The first loss is used to represent the loss between the coarse disparity value and the label disparity value after the discretization processing; the second loss is used to represent the loss between the matching disparity value and the label disparity value; and the third loss is used to represent the sum of the first loss and the second loss.

[0146] The coarse disparity value is an integer disparity value obtained by the disparity model output. For example, it can be 3.

[0147] The label disparity value can be 3.2, and the label disparity value after the discretization processing, i.e., the integer label disparity value, can be 3.

[0148] In a possible implementation, the first loss can be the loss between the coarse disparity value 3 and the label disparity value 3 after the discretization processing, i.e., the loss between the integer part of the matching disparity value and the integer part of the label disparity value.

[0149] In a possible implementation, the second loss can be the loss between the matching disparity value 3.1 and the label disparity value 3.2, i.e., the loss between the whole of the matching disparity value and the whole of the label disparity value.

[0150] In a possible implementation, the third loss can be the sum of the first loss and the second loss 0.1, i.e., the sum of the loss between the integer part of the matching disparity value and the integer part of the label disparity value, and the loss between the whole of the matching disparity value and the whole of the label disparity value.

[0151] In the foregoing embodiment, it is introduced that what kind of loss is included in the loss between the matching disparity value and the label disparity value. In another embodiment of the present application, it is introduced how to train the disparity matching model, Figure 10 The training process flowchart of the disparity matching model provided in the embodiment of the present application is described with reference to Figure 10 The flowchart includes the following steps:

[0152] P1, left-eye image, right-eye image.

[0153] It should be noted that the left-eye image and the right-eye image herein correspond to the sample image pair in the foregoing embodiment of the present application.

[0154] P2, feature extraction.

[0155] P3, left-eye feature map, right-eye feature map.

[0156] It should be noted that the left eye feature map and the right eye feature map herein correspond to the feature maps of the sample image pair in the foregoing embodiments of the present application.

[0157] P4, loss volume.

[0158] It should be noted that the loss volume herein corresponds to obtaining the matching degree corresponding to each of the plurality of candidate points in the foregoing embodiments of the present application.

[0159] P5, rough matching.

[0160] It should be noted that the rough matching herein corresponds to obtaining the rough disparity value in the matching disparity value in the foregoing embodiments of the present application.

[0161] P6, fine regression.

[0162] It should be noted that the fine regression herein corresponds to obtaining the corrected disparity value in the matching disparity value in the foregoing embodiments of the present application.

[0163] P7, disparity.

[0164] It should be noted that the disparity herein corresponds to the matching disparity value obtained by superimposing the rough disparity value and the corrected disparity value in the foregoing embodiments of the present application.

[0165] P8, disparity ground truth.

[0166] It should be noted that the disparity ground truth herein corresponds to the label disparity value in the foregoing embodiments of the present application.

[0167] P9, discretization.

[0168] It should be noted that the discretization herein corresponds to discretizing the label disparity value to obtain the discretized label disparity value in the foregoing embodiments of the present application.

[0169] P10, matching loss.

[0170] It should be noted that the matching loss herein corresponds to the first loss in the foregoing embodiments of the present application.

[0171] The detailed calculation process of the first loss is as follows:

[0172] First, for the input loss volume Vcost, the matching function (eg: argmax, argmin, softmax) is used to find the best matching mopt=softmax(Vcost);

[0173] It should be noted that the loss volume herein corresponds to the matching degree in the foregoing embodiments of the present application.

[0174] Second, the discretized disparity ground truth dispdiscrete = discrete(disp GT ).

[0175] It should be noted that the disparity true value disp GT here corresponds to the label disparity value in the foregoing embodiment of the present application, and the discrete disparity true value discrete(disp GT ) here corresponds to the label disparity value after the discrete processing in the foregoing embodiment of the present application.

[0176] Third, the loss loss match = categorical_cross_entropy(disp discrete ) is calculated.

[0177] It should be noted that the loss loss match here corresponds to the first loss in the foregoing embodiment of the present application.

[0178] P11, regression loss.

[0179] It should be noted that the regression loss here corresponds to the second loss in the foregoing embodiment of the present application.

[0180] The detailed calculation process of the second loss is as follows:

[0181] First, the input loss volume Vcost (N*S*C*H*W in shape and size) is reshaped into (N*S*C*H*W) and output regression (N*S*H*W in shape and size) using a convolution layer (conv2d, batch-normalization, activate) and converted into N*S*H*W, which is called disp pred .

[0182] It should be noted that the disp pred here corresponds to the matching disparity value obtained by superimposing the coarse disparity value and the corrected disparity value in the foregoing embodiment of the present application.

[0183] Second, the disparity true value (N*H*W) is converted into N*S*H*W as disp GT , and the regression loss loss regress = l1(disp GT , disp pred )

[0184] It should be noted that the disparity true value disp GT here corresponds to the label disparity value in the foregoing embodiment of the present application, and the disp pred here corresponds to the matching disparity value in the foregoing embodiment of the present application, and the regression loss lossregress loss

[0185] P12, total loss

[0186] It should be noted that the total loss here is equivalent to the third loss in the foregoing embodiments of the present application.

[0187] The detailed calculation process of the third loss is as follows:

[0188] loss total = a loss regress + (1-a) loss match .

[0189] Wherein, loss total is equivalent to the third loss in the foregoing embodiments of the present application, loss regress is equivalent to the second loss in the foregoing embodiments of the present application, and loss match is equivalent to the first loss in the foregoing embodiments of the present application, and a is a constant.

[0190] It should be noted that the remaining flowchart after deleting the above steps P8-P12 is the application process of the disparity matching model.

[0191] In the foregoing embodiments, the training process of a disparity matching model is introduced. In another embodiment of the present application, a disparity matching device is introduced, Figure 11 The structural schematic diagram of the disparity matching device provided in the embodiment of the present application is shown in the figure, and the device comprises:

[0192] The acquisition unit 1101 is configured to acquire an image pair collected by a binocular camera, and the image pair comprises a left eye image and a right eye image.

[0193] The feature extraction unit 1102 is configured to input the left eye image and the right eye image into a disparity matching model and perform feature extraction on the left eye image and the right eye image respectively to obtain a first feature map and a second feature map.

[0194] The processing unit 1103 is configured to perform epipolar constraint processing on the first feature map and the second feature map to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map and a matching degree corresponding to each of the plurality of candidate points; and the candidate point is a pixel point of the to-be-matched point on a symmetric epipolar line.

[0195] The target point screening unit 1104 is configured to screen a target point from the plurality of candidate points according to the matching degree.

[0196] The matching disparity value output unit 1105 is configured to obtain a matching disparity value output by the disparity matching model according to the target point, and the matching disparity value comprises a coarse disparity value and a corrected disparity value corresponding to the to-be-matched point.

[0197] The parallax matching point determination unit 1106 is configured to determine, according to the coarse parallax value and the corrected parallax value, a parallax matching point in the second feature map that matches the to-be-matched point in the first feature map.

[0198] In an embodiment, the processing unit 1103 is specifically configured to take the to-be-matched point in the first feature map as a search reference, take a preset offset as a search step, and obtain a plurality of similar pixel points in the second feature map that are on a symmetric epipolar line of the to-be-matched point; and determine a mapping point in the second feature map that has the same position as the to-be-matched point, and take a plurality of similar pixel points adjacent to the mapping point as candidate points.

[0199] In an embodiment, the matching parallax value output unit 1105 is configured to determine a mapping point in the second feature map that has the same position as the to-be-matched point; and take a pixel difference between the mapping point and the target point as the coarse parallax value.

[0200] In an embodiment, the matching parallax value output unit 1105 is configured to determine a corrected parallax value corresponding to the to-be-matched point according to a neighborhood point corresponding to the target point; perform mapping processing on pixel values in a neighborhood point interval to obtain a corrected parallax value in a preset numerical interval; and the preset numerical interval is a decimal interval.

[0201] In an embodiment, the parallax matching point determination unit 1106 is specifically configured to perform superposition processing on the coarse parallax value, the corrected parallax value, and a pixel value of the mapping point to obtain a parallax matching point in the second feature map for each to-be-matched point in the first feature map.

[0202] In an embodiment, the training process of the parallax matching model includes:

[0203] The sample image pair is input into the initial parallax matching model to perform feature extraction, to obtain a feature map of the sample image pair; the feature map of the sample image pair is processed to obtain a coarse parallax value and a corrected parallax value, and a matching parallax value is determined according to the coarse parallax value and the corrected parallax value; and the initial parallax matching model is iteratively trained based on a loss between the matching parallax value and a label parallax value, to obtain the parallax matching model.

[0204] The specific limitations of the parallax matching device can be referred to the limitations of the parallax matching method in the foregoing, which will not be repeated here. The modules of the parallax matching device described above can be realized by software, hardware, and combinations thereof, in whole or in part. The modules described above can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to call and execute the operations corresponding to the modules by the processor.

[0205] In an embodiment, a robot is provided. Figure 12The structural block diagram of the robot provided in the embodiments of the present application is shown in FIG. 1. The robot comprises a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the following steps: Figure 12 The robot comprises a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the following steps:

[0206] An image pair collected by a binocular camera is acquired. The image pair comprises a left-eye image and a right-eye image.

[0207] The left-eye image and the right-eye image are input into a disparity matching model for feature extraction to obtain a first feature map and a second feature map.

[0208] The first feature map and the second feature map are subjected to epipolar constraint processing to obtain a plurality of candidate points corresponding to at least one to-be-matched point in the first feature map in the second feature map, and a matching degree corresponding to each of the plurality of candidate points. The candidate point is a pixel point on a symmetric epipolar line of the to-be-matched point.

[0209] A target point is selected from the plurality of candidate points according to the matching degree.

[0210] A matching disparity value output by the disparity matching model is obtained according to the target point. The matching disparity value comprises a coarse disparity value and a corrected disparity value corresponding to the to-be-matched point.

[0211] A disparity matching point in the second feature map that matches the to-be-matched point in the first feature map is determined according to the coarse disparity value and the corrected disparity value.

[0212] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments can be included. Any reference to a memory, storage, database or other medium in each embodiment provided by the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory or an optical memory. The volatile memory can include a random access memory (RAM) or an external cache memory. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0213] The embodiments of the present application provide a computer program product, which comprises instructions, when the instructions are executed, cause the method as described in the embodiments of the present application to be performed. For example, the method can be executed by a computer program product. Figure 4 each step of the parallax matching method shown in the figure.

[0214] The technical features of the above embodiments can be combined in any manner. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, it should be considered that the combinations are within the scope of the present disclosure.

[0215] The above embodiments only express several implementation manners of the present application, the description is relatively specific and detailed, however, it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A disparity matching method, characterized in that, The method includes: Acquire image pairs captured by a binocular camera, the image pairs including a left-eye image and a right-eye image; The left and right eye images are input into the disparity matching model, and feature extraction is performed to obtain the first feature map and the second feature map. The first feature map and the second feature map are subjected to epipolar constraint processing to obtain multiple candidate points in the second feature map corresponding to at least one point to be matched in the first feature map, and the matching degree of each of the multiple candidate points; the candidate points are the pixels of the point to be matched on the symmetrical epipolar line. The target point is obtained by filtering from the plurality of candidate points based on the matching degree; The matching disparity value output by the disparity matching model is obtained based on the target point. The matching disparity value includes the coarse disparity value and the corrected disparity value corresponding to the point to be matched. Based on the coarse disparity value and the corrected disparity value, determine the disparity matching point in the second feature map that matches the point to be matched in the first feature map; The specific implementation of obtaining the matching disparity value output by the disparity matching model based on the target point includes: In the second feature map, a mapping point with the same position as the point to be matched is determined; the pixel difference between the mapping point and the target point is used as a coarse disparity value; the corrected disparity value corresponding to the point to be matched is determined based on the neighboring points corresponding to the target point; the pixel values ​​within the neighboring point interval are mapped to obtain the corrected disparity value within a preset value interval; the preset value interval is a decimal interval; the neighboring point interval is the interval composed of the target point and its adjacent points.

2. The method according to claim 1, characterized in that, The epipolar constraint processing of the first feature map and the second feature map yields multiple candidate points in the second feature map corresponding to at least one point to be matched in the first feature map, including: Using the point to be matched in the first feature map as the search reference and the preset offset as the search step size, multiple similar pixel points in the second feature map that are on the symmetrical epipolar line with the point to be matched are obtained. In the second feature map, a mapping point with the same position as the point to be matched is determined, and multiple similar pixels adjacent to the mapping point are selected as candidate points.

3. The method according to any one of claims 1-2, characterized in that, The step of determining the disparity matching point in the second feature map that matches the point to be matched in the first feature map based on the coarse disparity value and the corrected disparity value includes: The coarse disparity value, the corrected disparity value, and the pixel value of the mapped point are superimposed to obtain the disparity matching point of each point to be matched in the first feature map in the second feature map.

4. The method according to claim 1, characterized in that, The training process of the disparity matching model includes: The sample images are input to the initial disparity matching model to extract features, thereby obtaining the feature maps of the sample image pairs. After processing the feature maps of the sample image pairs, coarse disparity values ​​and corrected disparity values ​​are obtained, and matching disparity values ​​are determined based on the coarse disparity values ​​and the corrected disparity values. Based on the loss between the matched disparity value and the label disparity value, the initial disparity matching model is iteratively trained to obtain the disparity matching model.

5. The method according to claim 4, characterized in that, The loss based on the match disparity value and the label disparity value includes a first loss, a second loss, and a third loss; The first loss is used to characterize the loss between the coarse disparity value and the discretized label disparity value; the second loss is used to characterize the loss between the matched disparity value and the label disparity value; and the third loss is used to characterize the sum of the first loss and the second loss.

6. A parallax matching device, characterized in that, The apparatus for implementing the method as described in any one of claims 1-5 includes: An acquisition unit is used to acquire image pairs captured by a binocular camera, the image pairs including a left-eye image and a right-eye image; The feature extraction unit is used to input the left eye image and the right eye image into the disparity matching model, perform feature extraction respectively, and obtain a first feature map and a second feature map. The processing unit is configured to perform epipolar constraint processing on the first feature map and the second feature map to obtain multiple candidate points in the second feature map corresponding to at least one point to be matched in the first feature map, and the matching degree corresponding to each of the multiple candidate points; the candidate points are the pixels of the point to be matched on the symmetrical epipolar line. A target point filtering unit is used to filter target points from the plurality of candidate points according to the matching degree; The matching disparity value output unit is used to obtain the matching disparity value output by the disparity matching model based on the target point. The matching disparity value includes the coarse disparity value and the corrected disparity value corresponding to the point to be matched. The disparity matching point determination unit is used to determine, based on the coarse disparity value and the corrected disparity value, a disparity matching point in the second feature map that matches the point to be matched in the first feature map.

7. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the disparity matching method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the disparity matching method as described in any one of claims 1-5.

Images