Obstacle detection method, intelligent driving control method, device, medium and equipment

Abstract

The embodiments of the present disclosure disclose an obstacle detection method and device, an electronic device, a computer-readable storage medium, and a computer program, wherein the obstacle detection method includes: obtaining a first disparity map of an environment image, wherein the environment image is an image representing the spatial environment information of a smart device during movement; in the first disparity map, determining a plurality of obstacle pixel areas; clustering the plurality of obstacle pixel areas to obtain at least one cluster; and determining an obstacle detection result based on obstacle pixel areas belonging to the same cluster.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and more particularly to an obstacle detection method, an obstacle detection device, an intelligent driving control method, an intelligent driving control device, an electronic device, a computer-readable storage medium, and a computer program. Background Art

[0002] In the field of computer vision technology, perception technology is usually used to perceive obstacles in the outside world. That is, perception technology includes obstacle detection.

[0003] The perception results of the perception technology are usually provided to the decision-making layer, so that the decision-making layer can make decisions based on the perception results. For example, in an intelligent driving system, the perception layer provides the road information of the vehicle and the obstacle information around the vehicle to the decision-making layer, so that the decision-making layer can make driving decisions to avoid obstacles and ensure the safe driving of the vehicle. In related technologies, the types of obstacles are generally pre-defined, such as pedestrians, vehicles, non-motor vehicles and other obstacles with inherent shapes, textures, and colors, and then relevant detection algorithms are used to detect obstacles of pre-defined types. Summary of the invention

[0004] The disclosed embodiments provide a technical solution for obstacle detection and intelligent driving control.

[0005] According to one aspect of the embodiments of the present disclosure, there is provided an obstacle detection method, comprising: obtaining a first disparity map of an environment image, wherein the environment image is an image representing spatial environment information in which a smart device is located during movement; determining a plurality of obstacle pixel areas in the first disparity map of the environment image; clustering the plurality of obstacle pixel areas to obtain at least one cluster; and determining an obstacle detection result based on obstacle pixel areas belonging to the same cluster.

[0006] In one embodiment of the present disclosure, the environmental image includes a monocular image; after obtaining a first disparity map of the monocular image, it also includes: mirroring the monocular image to obtain a first mirror image, and obtaining a disparity map of the first mirror image; according to the disparity map of the first mirror image, performing disparity adjustment on the first disparity map of the monocular image to obtain a first disparity map after disparity adjustment; determining a plurality of obstacle pixel areas in the first disparity map of the environmental image, includes: determining a plurality of obstacle pixel areas in the first disparity map after disparity adjustment.

[0007] In yet another embodiment of the present disclosure, the method of performing disparity adjustment on the first disparity map of the monocular image according to the disparity map of the first mirror image to obtain the first disparity map after disparity adjustment includes: performing mirroring processing on the disparity map of the first mirror image to obtain a second mirror image; performing disparity adjustment on the first disparity map according to a weight distribution map of the first disparity map and a weight distribution map of the second mirror image to obtain the first disparity map after disparity adjustment; wherein the weight distribution map of the first disparity map includes weight values ​​corresponding to each of multiple disparity values ​​in the first disparity map; and the weight distribution map of the second mirror image includes weights corresponding to each of multiple disparity values ​​in the second mirror image.

[0008] In another embodiment of the present disclosure, the weight distribution map includes: a first weight distribution map, and / or a second weight distribution map; the first weight distribution map is a weight distribution map uniformly set for multiple environmental images; the second weight distribution map is a weight distribution map set separately for different environmental images.

[0009] In yet another embodiment of the present disclosure, the first weight distribution map includes at least two areas in left and right columns, and different areas have different weight values.

[0010] In another embodiment of the present disclosure, when the monocular image is a left-eye image to be processed: for any two areas in the first weight distribution map of the first disparity map, the weight value of the area on the right is not less than the weight value of the area on the left; for any two areas in the first weight distribution map of the second mirror image, the weight value of the area on the right is not less than the weight value of the area on the left.

[0011] In still another embodiment of the present disclosure, for at least one area in the first weight distribution map of the first disparity map, the weight value of the left part of the area is not greater than the weight value of the right part of the area; for at least one area in the first weight distribution map of the second mirror image, the weight value of the left part of the area is not greater than the weight value of the right part of the area.

[0012] In another embodiment of the present disclosure, when the monocular image is a right-eye image to be processed: for any two areas in the first weight distribution map of the first disparity map, the weight value of the area on the left is not less than the weight value of the area on the right; for any two areas in the first weight distribution map of the second mirror image, the weight value of the area on the left is not less than the weight value of the area on the right.

[0013] In still another embodiment of the present disclosure, for at least one area in the first weight distribution map of the first disparity map, the weight value of the right part of the area is not greater than the weight value of the left part of the area; for at least one area in the first weight distribution map of the second mirror image, the weight value of the right part of the area is not greater than the weight value of the left part of the area.

[0014] In another embodiment of the present disclosure, the second weight distribution map of the first disparity map is set in a manner including: mirroring the first disparity map to form a mirror disparity map; and setting weight values ​​in the second weight distribution map of the first disparity map according to disparity values ​​in the mirror disparity map of the first disparity map.

[0015] In still another embodiment of the present disclosure, setting the weight value in the second weight distribution map of the first disparity map according to the disparity value in the mirror disparity map of the first disparity map includes: for a pixel point at any position in the mirror disparity map, when the disparity value of the pixel point at the position satisfies a first predetermined condition, setting the weight value of the pixel point at the position in the second weight distribution map of the first disparity map to a first value.

[0016] In another embodiment of the present disclosure, when the disparity value of the pixel point does not satisfy the first predetermined condition, the method further includes: setting the weight value of the pixel point at the position in the second weight distribution map of the first disparity map to a second value; wherein the first value is greater than the second value.

[0017] In yet another embodiment of the present disclosure, the first predetermined condition includes: the disparity value of the pixel point at the position is greater than the first reference value of the pixel point at the position;

[0018] The first reference value of the pixel at the position is set according to the disparity value of the pixel at the position in the first disparity map and a constant value greater than zero.

[0019] In yet another embodiment of the present disclosure, the second weight distribution map of the second mirror image is set in a manner including: setting weight values ​​in the second weight distribution map of the second mirror image according to disparity values ​​in the first disparity map.

[0020] In yet another embodiment of the present disclosure, setting the weight value in the second weight distribution map of the second mirror image according to the disparity value in the first disparity map includes: for a pixel point at any position in the second mirror image, if the disparity value of the pixel point at that position in the first disparity map satisfies a second predetermined condition, setting the weight value of the pixel point at that position in the second weight distribution map of the second mirror image to a third value.

[0021] In yet another embodiment of the present disclosure, when the disparity value of the pixel point at the position in the first disparity map does not satisfy the second predetermined condition, the method further includes: setting the weight value of the pixel point at the position in the second weight distribution map of the second mirror image to a fourth value; wherein the third value is greater than the fourth value.

[0022] In another embodiment of the present disclosure, the second predetermined condition includes: the disparity value of the pixel point at the position in the first disparity map is greater than the second reference value of the pixel point at the position; wherein the second reference value of the pixel point at the position is set based on the disparity value of the pixel point at the position in the mirror disparity map of the first disparity map and a constant value greater than zero.

[0023] In yet another embodiment of the present disclosure, the first disparity map is disparity-adjusted according to the weight distribution map of the first disparity map and the weight distribution map of the second mirror map to obtain the first disparity map after disparity adjustment, including: adjusting the disparity value in the first disparity map according to the first weight distribution map and the second weight distribution map of the first disparity map; adjusting the disparity value in the second mirror map according to the first weight distribution map and the second weight distribution map of the second mirror map; and merging the first disparity map after disparity adjustment and the second mirror map after parallax value adjustment to finally obtain the first disparity map after parallax adjustment.

[0024] In one embodiment of the present disclosure, the environment image includes a monocular image; the obtaining of the first disparity map of the monocular image includes: performing disparity analysis processing on the monocular image using a convolutional neural network, and obtaining the first disparity map of the monocular image based on the output of the convolutional neural network; wherein the convolutional neural network is obtained by training using binocular image samples.

[0025] In another embodiment of the present disclosure, the training process of the convolutional neural network includes: inputting one of the binocular image samples into the convolutional neural network to be trained, performing disparity analysis processing through the convolutional neural network, and obtaining a disparity map of the left-eye image sample and a disparity map of the right-eye image sample based on the output of the convolutional neural network; reconstructing the right-eye image according to the disparity map of the left-eye image sample and the right-eye image sample; reconstructing the left-eye image according to the disparity map of the right-eye image sample and the left-eye image sample; and adjusting the network parameters of the convolutional neural network according to the difference between the reconstructed left-eye image and the left-eye image sample, and the difference between the reconstructed right-eye image and the right-eye image sample.

[0026] In another embodiment of the present disclosure, the determining of multiple obstacle pixel areas in the first disparity map of the environment image includes: performing edge detection on the first disparity map of the environment image to obtain obstacle edge information; determining the obstacle area in the first disparity map of the environment image; and determining multiple obstacle pixel columnar areas in the obstacle area of ​​the first disparity map based on the obstacle edge information.

[0027] In another embodiment of the present disclosure, determining the obstacle area in the first disparity map of the environmental image includes: performing statistical processing on the disparity values ​​of each row of pixels in the first disparity map to obtain statistical information on the disparity values ​​of each row of pixels; determining a statistical disparity map based on the statistical information on the disparity values ​​of each row of pixels; performing a first straight line fitting process on the statistical disparity map, and determining a ground area and a non-ground area based on a result of the first straight line fitting process; and determining an obstacle area based on the non-ground area.

[0028] In still another embodiment of the present disclosure, the non-ground area includes: a first area above the ground; or, the non-ground area includes: a first area above the ground and a second area below the ground.

[0029] In still another embodiment of the present disclosure, determining the obstacle area based on the non-ground area includes: performing a second straight line fitting process on the statistical disparity map, and determining a first target area in the first area whose height above the ground is less than a first predetermined height value based on a result of the second straight line fitting process, the first target area being the obstacle area; and in the case where there is a second area below the ground in the non-ground area, determining a second target area in the second area whose height below the ground is greater than a second predetermined height value, the second target area being the obstacle area.

[0030] In another embodiment of the present disclosure, based on the obstacle edge information, multiple obstacle pixel columnar areas are determined in the obstacle area of ​​the first disparity map, including: setting the disparity values ​​of the pixel points in the non-obstacle area of ​​the first disparity map and the disparity values ​​of the pixel points at the obstacle edge information to predetermined values; taking N pixel points in the column direction of the first disparity map as traversal units, traversing the disparity values ​​of N pixel points on each row starting from a set row of the first disparity map, and determining a target row where the disparity value of the pixel point jumps between the predetermined value and the non-predetermined value; N is a positive integer; taking N pixel points in the column direction as the column width and the determined target row as the boundary of the obstacle pixel columnar area in the row direction, the obstacle pixel columnar area in the obstacle area is determined.

[0031] In yet another embodiment of the present disclosure, the obstacle pixel area includes an obstacle pixel column area; the clustering process for the plurality of obstacle pixel areas includes: determining spatial position information of the plurality of obstacle pixel column areas; and clustering the plurality of obstacle pixel column areas according to the spatial position information of the plurality of obstacle pixel column areas.

[0032] In yet another embodiment of the present disclosure, determining the spatial position information of the plurality of obstacle pixel columnar regions includes: for any obstacle pixel columnar region, determining attribute information of the obstacle pixel columnar region based on pixels included in the obstacle pixel columnar region, and determining the spatial position information of the obstacle pixel columnar region based on the attribute information of the obstacle pixel columnar region.

[0033] In yet another embodiment of the present disclosure, the attribute information of the obstacle pixel column region includes at least one of: pixel column region bottom information, pixel column region top information, pixel column region disparity value, and pixel column region column information.

[0034] In yet another embodiment of the present disclosure, the spatial position information of the obstacle pixel column region includes: coordinates of the obstacle pixel column region on a horizontal coordinate axis and coordinates of the obstacle pixel column region on a depth coordinate axis.

[0035] In another embodiment of the present disclosure, the spatial position information of the obstacle pixel column area also includes: the highest point coordinates of the obstacle pixel column area on the vertical coordinate axis, and the lowest point coordinates of the obstacle pixel column area on the vertical coordinate axis; the highest point coordinates and the lowest point coordinates are used to determine the obstacle height.

[0036] In another embodiment of the present disclosure, the obstacle pixel area includes: an obstacle pixel columnar area; determining the obstacle detection result based on the obstacle pixel areas belonging to the same cluster includes: determining the obstacle detection frame in the environment image based on the spatial position information of the obstacle pixel columnar areas belonging to the same cluster; and / or determining the spatial position information of the obstacle based on the spatial position information of the obstacle pixel columnar areas belonging to the same cluster.

[0037] In yet another embodiment of the present disclosure, determining the spatial position information of the obstacle based on the spatial position information of the obstacle pixel columnar regions belonging to the same cluster includes: determining the distance between the multiple obstacle pixel columnar regions and a camera device that generates the environment image based on the spatial position information of the multiple obstacle pixel columnar regions belonging to the same cluster; and determining the spatial position information of the obstacle based on the spatial position information of the obstacle pixel columnar region closest to the camera device.

[0038] According to another aspect of the embodiments of the present disclosure, there is provided an intelligent driving control method, the method comprising: acquiring an environment image of the intelligent driving device during movement by an image acquisition device provided on the intelligent driving device; performing obstacle detection on the acquired environment image by using the obstacle detection method of any of the above embodiments to determine an obstacle detection result; and generating and outputting a control instruction according to the obstacle detection result.

[0039] According to another aspect of the embodiments of the present disclosure, an obstacle detection device is provided, including: an acquisition module, used to acquire a first disparity map of an environment image, wherein the environment image is an image representing spatial environment information in which a smart device is located during movement; a first determination module, used to determine a plurality of obstacle pixel areas in the first disparity map of the environment image; a clustering module, used to cluster the plurality of obstacle pixel areas to obtain at least one cluster; and a second determination module, used to determine an obstacle detection result based on obstacle pixel areas belonging to the same cluster.

[0040] In one embodiment of the present disclosure, the acquisition module also includes: a second submodule, used to obtain a first mirror image after mirroring the monocular image, and obtain a disparity map of the first mirror image; a third submodule, used to perform disparity adjustment on the first disparity map of the monocular image according to the disparity map of the first mirror image, and obtain the first disparity map after disparity adjustment; the first determination module is further used to determine multiple obstacle pixel areas in the first disparity map after disparity adjustment.

[0041] In another embodiment of the present disclosure, the third submodule includes: a first unit, used to obtain a second mirror image after mirroring the disparity map of the first mirror image; a second unit, used to perform disparity adjustment on the first disparity map according to the weight distribution map of the first disparity map and the weight distribution map of the second mirror image, to obtain the first disparity map after disparity adjustment; wherein the weight distribution map of the first disparity map includes weight values ​​corresponding to each of multiple disparity values ​​in the first disparity map; and the weight distribution map of the second mirror image includes weights corresponding to each of multiple disparity values ​​in the second mirror image.

[0042] In another embodiment of the present disclosure, the weight distribution map includes: a first weight distribution map, and / or a second weight distribution map; the first weight distribution map is a weight distribution map uniformly set for multiple environmental images; the second weight distribution map is a weight distribution map set separately for different environmental images.

[0043] In yet another embodiment of the present disclosure, the first weight distribution map includes at least two areas in left and right columns, and different areas have different weight values.

[0044] In another embodiment of the present disclosure, when the monocular image is a left-eye image to be processed: for any two areas in the first weight distribution map of the first disparity map, the weight value of the area on the right is not less than the weight value of the area on the left; for any two areas in the first weight distribution map of the second mirror image, the weight value of the area on the right is not less than the weight value of the area on the left.

[0045] In still another embodiment of the present disclosure, for at least one area in the first weight distribution map of the first disparity map, the weight value of the left part of the area is not greater than the weight value of the right part of the area; for at least one area in the first weight distribution map of the second mirror image, the weight value of the left part of the area is not greater than the weight value of the right part of the area.

[0046] In another embodiment of the present disclosure, when the monocular image is a right-eye image to be processed: for any two areas in the first weight distribution map of the first disparity map, the weight value of the area on the left is not less than the weight value of the area on the right; for any two areas in the first weight distribution map of the second mirror image, the weight value of the area on the left is not less than the weight value of the area on the right.

[0047] In still another embodiment of the present disclosure, for at least one area in the first weight distribution map of the first disparity map, the weight value of the right part of the area is not greater than the weight value of the left part of the area; for at least one area in the first weight distribution map of the second mirror image, the weight value of the right part of the area is not greater than the weight value of the left part of the area.

[0048] In another embodiment of the present disclosure, the third submodule also includes: a third unit for setting a second weight distribution map of the first disparity map; the third unit performs mirroring on the first disparity map to form a mirror disparity map; and sets the weight value in the second weight distribution map of the first disparity map according to the disparity value in the mirror disparity map of the first disparity map.

[0049] In another embodiment of the present disclosure, the third unit is further used to, for a pixel point at any position in the mirror disparity map, set the weight value of the pixel point at this position in the second weight distribution map of the first disparity map to a first value when the disparity value of the pixel point at this position satisfies a first predetermined condition.

[0050] In another embodiment of the present disclosure, the third unit is further used to set the weight value of the pixel point at the position in the second weight distribution map of the first disparity map to a second value when the disparity value of the pixel point does not satisfy the first predetermined condition; wherein the first value is greater than the second value.

[0051] In another embodiment of the present disclosure, the first predetermined condition includes: the disparity value of the pixel point at the position is greater than the first reference value of the pixel point at the position; wherein the first reference value of the pixel point at the position is set based on the disparity value of the pixel point at the position in the first disparity map and a constant value greater than zero.

[0052] In another embodiment of the present disclosure, the third submodule further includes: a fourth unit for setting a second weight distribution map of the second mirror image; the fourth unit sets the weight value in the second weight distribution map of the second mirror image according to the disparity value in the first disparity map.

[0053] In another embodiment of the present disclosure, the fourth unit is further used for: for a pixel point at any position in the second mirror image, if the disparity value of the pixel point at that position in the first disparity map satisfies a second predetermined condition, the weight value of the pixel point at that position in the second weight distribution map of the second mirror image is set to a third value.

[0054] In yet another embodiment of the present disclosure, the fourth unit is further used to set the weight value of the pixel point at the position in the second weight distribution map of the second mirror image to a fourth value when the disparity value of the pixel point at the position in the first disparity map does not satisfy a second predetermined condition; wherein the third value is greater than the fourth value.

[0055] In another embodiment of the present disclosure, the second predetermined condition includes: the disparity value of the pixel point at the position in the first disparity map is greater than the second reference value of the pixel point at the position; wherein the second reference value of the pixel point at the position is set based on the disparity value of the pixel point at the position in the mirror disparity map of the first disparity map and a constant value greater than zero.

[0056] In another embodiment of the present disclosure, the second unit is further used to: adjust the disparity value in the first disparity image according to the first weight distribution map and the second weight distribution map of the first disparity image; adjust the disparity value in the second mirror image according to the first weight distribution map and the second weight distribution map of the second mirror image; merge the first disparity image after disparity adjustment and the second mirror image after disparity value adjustment, and finally obtain the first disparity image after parallax adjustment.

[0057] In one embodiment of the present disclosure, the environmental image includes a monocular image; the acquisition module includes: a first submodule, used to perform disparity analysis on the monocular image using a convolutional neural network, and obtain a first disparity map of the monocular image based on the output of the convolutional neural network; wherein the convolutional neural network is obtained by training using binocular image samples.

[0058] In another embodiment of the present disclosure, the device also includes: a training module for training a convolutional neural network, and the training module is further used to: input one of the binocular image samples into the convolutional neural network to be trained, perform disparity analysis processing through the convolutional neural network, and obtain a disparity map of the left-eye image sample and a disparity map of the right-eye image sample based on the output of the convolutional neural network; reconstruct the right-eye image according to the disparity map of the left-eye image sample and the right-eye image sample; reconstruct the left-eye image according to the disparity map of the right-eye image sample and the left-eye image sample; adjust the network parameters of the convolutional neural network according to the difference between the reconstructed left-eye image and the left-eye image sample, and the difference between the reconstructed right-eye image and the right-eye image sample.

[0059] In another embodiment of the present disclosure, the first determination module includes: a fourth submodule, used to perform edge detection on the first disparity map of the environment image to obtain obstacle edge information; a fifth submodule, used to determine the obstacle area in the first disparity map of the environment image; and a sixth submodule, used to determine a plurality of obstacle pixel columnar areas in the obstacle area of ​​the first disparity map based on the obstacle edge information.

[0060] In another embodiment of the present disclosure, the fifth submodule includes: a fifth unit, used to perform statistical processing on the disparity values ​​of each row of pixels in the first disparity map to obtain statistical information on the disparity values ​​of each row of pixels; a sixth unit, used to determine the statistical disparity map based on the statistical information on the disparity values ​​of each row of pixels; a seventh unit, used to perform a first straight line fitting process on the statistical disparity map, and determine the ground area and the non-ground area according to the result of the first straight line fitting process; and an eighth unit, used to determine the obstacle area based on the non-ground area.

[0061] In still another embodiment of the present disclosure, the non-ground area includes: a first area above the ground; or, the non-ground area includes: a first area above the ground and a second area below the ground.

[0062] In another embodiment of the present disclosure, the eighth unit is further used to perform a second straight-line fitting process on the statistical disparity map, and based on the result of the second straight-line fitting process, determine a first target area in the first area whose height above the ground is less than a first predetermined height value, and the first target area is an obstacle area; in the case where there is a second area below the ground in the non-ground area, determine a second target area in the second area whose height below the ground is greater than a second predetermined height value, and the second target area is an obstacle area.

[0063] In another embodiment of the present disclosure, the sixth submodule is further used to: set the disparity values ​​of the pixel points in the non-obstacle area in the first disparity map and the disparity values ​​of the pixel points at the obstacle edge information to predetermined values; take the N pixel points in the column direction of the first disparity map as traversal units, traverse the disparity values ​​of the N pixel points on each row starting from the set row of the first disparity map, and determine the target row where the disparity value of the pixel point jumps between the predetermined value and the non-predetermined value; N is a positive integer; take the N pixel points in the column direction as the column width and the determined target row as the boundary of the obstacle pixel columnar area in the row direction to determine the obstacle pixel columnar area in the obstacle area.

[0064] In another embodiment of the present disclosure, the obstacle pixel area includes an obstacle pixel column area; the clustering module includes: a seventh submodule, used to determine the spatial position information of the multiple obstacle pixel column areas; an eighth submodule, used to perform clustering processing on the multiple obstacle pixel column areas according to the spatial position information of the multiple obstacle pixel column areas.

[0065] In another embodiment of the present disclosure, the eighth submodule is further used to: for any obstacle pixel column area, determine the attribute information of the obstacle pixel column area according to the pixels included in the obstacle pixel column area, and determine the spatial position information of the obstacle pixel column area according to the attribute information of the obstacle pixel column area.

[0066] In yet another embodiment of the present disclosure, the attribute information of the obstacle pixel column region includes at least one of: pixel column region bottom information, pixel column region top information, pixel column region disparity value, and pixel column region column information.

[0067] In yet another embodiment of the present disclosure, the spatial position information of the obstacle pixel column region includes: coordinates of the obstacle pixel column region on a horizontal coordinate axis and coordinates of the obstacle pixel column region on a depth coordinate axis.

[0068] In another embodiment of the present disclosure, the spatial position information of the obstacle pixel column area also includes: the highest point coordinates of the obstacle pixel column area on the vertical coordinate axis, and the lowest point coordinates of the obstacle pixel column area on the vertical coordinate axis; the highest point coordinates and the lowest point coordinates are used to determine the obstacle height.

[0069] In yet another embodiment of the present disclosure, the obstacle pixel area includes: an obstacle pixel column area; and the second determination module includes:

[0070] The ninth submodule is used to determine the obstacle detection frame in the environmental image based on the spatial position information of the obstacle pixel columnar area belonging to the same cluster; and / or, the tenth submodule is used to determine the spatial position information of the obstacle based on the spatial position information of the obstacle pixel columnar area belonging to the same cluster.

[0071] In another embodiment of the present disclosure, the tenth submodule is further used to: determine the distance between the multiple obstacle pixel column areas and the camera device that generates the environment image based on the spatial position information of the multiple obstacle pixel column areas belonging to the same cluster; determine the spatial position information of the obstacle based on the spatial position information of the obstacle pixel column area closest to the camera device.

[0072] According to another aspect of the embodiments of the present disclosure, there is provided an intelligent driving control device, which includes: an acquisition module, used to acquire an environmental image of the intelligent device during movement through an image acquisition device provided on the intelligent driving device; using the above-mentioned obstacle detection device to perform obstacle detection on the environmental image and determine the obstacle detection result; and a control module, used to generate and output a control instruction according to the obstacle detection result.

[0073] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, any method embodiment of the present disclosure is implemented.

[0074] According to another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any method embodiment of the present disclosure is implemented.

[0075] According to another aspect of the embodiments of the present disclosure, there is provided a computer program, comprising computer instructions, which implement any method embodiment of the present disclosure when the computer instructions are executed in a processor of a device.

[0076] Based on the obstacle detection method and device, intelligent driving control method and device, electronic device, computer-readable storage medium and computer program provided by the present disclosure, the present disclosure can determine multiple obstacle pixel areas from the first disparity map of the environmental image, and obtain obstacle detection results by clustering the multiple obstacle pixel areas. The detection method adopted by the present disclosure does not need to predefine the obstacles to be detected, and does not use the predefined information such as the texture, color, shape, and category of the obstacles. It can directly detect obstacles based on clustering the obstacle areas. Moreover, the detected obstacles are not limited to certain predefined obstacles, and can detect various obstacles (which can be called generic obstacles in the present disclosure) in the surrounding space environment that may hinder the movement of the smart device, thereby realizing generic obstacle detection.

[0077] Compared with the method of detecting predefined types of obstacles in related technologies, the technical solution provided by the present disclosure is a more general obstacle detection solution, which can be applied to the detection of general types of obstacles, and is conducive to the detection of various types of obstacles in real environments. In addition, for intelligent driving equipment, the technical solution provided by the present disclosure can realize the detection of various random obstacles that may appear during driving, and then output the control instructions of the driving process based on the detection results, which is conducive to improving the safety of intelligent driving of vehicles.

[0078] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0080] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0081] Figure 1 A flowchart of an embodiment of the obstacle detection method disclosed in the present invention;

[0082] Figure 2 A schematic diagram of an embodiment of an environment image disclosed in the present invention;

[0083] Figure 3 for Figure 2 The first parallax Figure 1 A schematic diagram of an embodiment;

[0084] Figure 4 The first parallax of the present disclosure Figure 1 A schematic diagram of an embodiment;

[0085] Figure 5 A schematic diagram of an embodiment of a convolutional neural network disclosed in the present invention;

[0086] Figure 6 is the first weight distribution of the first disparity map disclosed in the present invention Figure 1 A schematic diagram of an embodiment;

[0087] Figure 7 is a schematic diagram of another embodiment of the first weight distribution map of the first disparity map disclosed in the present invention;

[0088] Figure 8 is the second weight distribution of the first disparity map of the present disclosure Figure 1 A schematic diagram of an embodiment;

[0089] Fig. 9 The second mirror image of this disclosure Figure 1 A schematic diagram of an embodiment;

[0090] Fig.10 for Fig. 9 The second weight distribution of the second mirror image shown Figure 1 A schematic diagram of an embodiment;

[0091] Fig.11 A schematic diagram of an implementation method of optimizing and adjusting a disparity map of a monocular image according to the present disclosure;

[0092] Fig.12 A schematic diagram of an implementation of obstacle edge information in a first disparity map of an environment image disclosed in the present invention;

[0093] Fig.13 The statistical parallax of the present disclosure Figure 1 Schematic diagram of implementation method;

[0094] Fig.14 A schematic diagram of an implementation of forming a statistical disparity map according to the present disclosure;

[0095] Fig.15 A schematic diagram of an embodiment of straight line fitting disclosed in the present invention;

[0096] Fig.16 A schematic diagram of a ground area and a non-ground area of ​​the present disclosure;

[0097] Fig.17 A schematic diagram of an implementation of a coordinate system established for the present disclosure;

[0098] Fig.18 It is a schematic diagram of two areas included in the first area above the ground in the present disclosure;

[0099] Fig.19A schematic diagram of an embodiment of forming an obstacle pixel column region disclosed in the present invention;

[0100] Fig. 20 A schematic diagram of an implementation of clustering obstacle pixel columnar regions according to the present disclosure;

[0101] Fig.21 A schematic diagram of an implementation of forming an obstacle detection frame disclosed in the present invention;

[0102] Fig. 22 A flowchart of an embodiment of a convolutional neural network training method disclosed in the present invention;

[0103] Fig.23 A flowchart of an implementation of the intelligent driving control method disclosed herein;

[0104] Fig.24 It is a structural schematic diagram of an embodiment of an obstacle detection device disclosed in the present invention;

[0105] Fig.25 This is a flow chart of an implementation of the intelligent driving control device disclosed herein;

[0106] Fig.26 A block diagram of an exemplary device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0107] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0108] At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. The techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be considered part of the specification. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0109] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems.

[0110] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. In general, program modules can include routines, programs, object programs, components, logic and data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0111] Exemplary Embodiments

[0112] Figure 1 FIG. 1 is a flow chart of an embodiment of the obstacle detection method disclosed in the present invention. Figure 1 As shown, the method of this embodiment includes steps: S100, S110, S120 and S130. Each step is described in detail below.

[0113] S100: Obtain a first disparity map of an environment image, wherein the environment image is an image representing spatial environment information in which the smart device is located during movement.

[0114] Exemplarily, the intelligent device is, for example, an intelligent driving device (such as an autonomous vehicle), an intelligent flight device (such as a drone), an intelligent robot, etc. The environmental image is, for example, an image representing the road space environmental information of the intelligent driving device or the intelligent robot during movement, or an image representing the spatial environmental information of the intelligent flight device during flight. Of course, the intelligent devices and environmental images in the present disclosure are not limited to the above examples, and the present disclosure does not limit this.

[0115] In the present disclosure, obstacles in the environmental image are detected, wherein any object in the surrounding space environment where the smart device is located that may cause obstacles to the movement process may fall into the obstacle detection range and be regarded as an obstacle detection object. For example, when the smart driving device is driving, objects such as stones, animals, and fallen goods may appear on the road surface. These objects have no specific shape, texture, color, or category, and they are very different from each other, and they can all be regarded as obstacles. In the present disclosure, any object that may cause obstacles in the movement process is called a generic obstacle.

[0116] In an optional example, the first disparity map of the present disclosure is used to describe the disparity of the environment image. Disparity can be considered to refer to the difference in the position of the target object when observing the same target object from two points at a certain distance. An example of an environment image is Figure 2 shown. Figure 2 An example of a first disparity map of the environment image is shown in Figure 3 Optionally, the first disparity map of the environment image in the present disclosure can also be expressed as follows Figure 4 The form shown. Figure 4 The numbers in (such as 0, 1, 2, 3, 4 and 5, etc.) represent the disparity of the pixel at the position (x, y) in the environment image. It should be noted that Figure 4 A complete disparity map is not shown.

[0117] In an optional example, the environment image in the present disclosure may be a monocular image or a binocular image to be processed. A monocular image is usually an image obtained by shooting with a monocular camera device. A binocular image to be processed is usually an image obtained by shooting with a binocular camera device. Optionally, the monocular image and the binocular image to be processed in the present disclosure may be photos or pictures, or may be video frames in a video.

[0118] When the environment image is a monocular image, the present disclosure can realize obstacle detection without setting up a binocular camera device, thereby helping to reduce the obstacle detection cost.

[0119] In an optional embodiment, when the environment image is a monocular image, the present disclosure can use a convolutional neural network that has been successfully trained in advance to obtain a first disparity map of the monocular image. For example, the monocular image is input into a convolutional neural network, and the monocular image is subjected to disparity analysis processing via the convolutional neural network. The convolutional neural network outputs the disparity analysis processing result, so that the present disclosure can obtain the first disparity map of the monocular image based on the disparity analysis processing result. By using a convolutional neural network to obtain the first disparity map of the monocular image, the first disparity map can be obtained without using two images for pixel-by-pixel disparity calculation and without the need for camera device calibration. This is conducive to improving the convenience and real-time performance of obtaining the first disparity map.

[0120] In an optional example, the convolutional neural network in the present disclosure generally includes but is not limited to: multiple convolutional layers (Conv) and multiple deconvolutional layers (Deconv). The convolutional neural network of the present disclosure can be divided into two parts, namely, an encoding part and a decoding part. The monocular image (such as Figure 2 The monocular image shown in FIG. 1 is encoded by the encoding part (i.e., feature extraction), and the encoding result of the encoding part is provided to the decoding part, which decodes the encoding result and outputs the decoding result. The present invention can obtain the first disparity map of the monocular image (such as FIG. 1 ) according to the decoding result output by the convolutional neural network. Figure 3 The first disparity map shown).

[0121] Optionally, the encoding part of the convolutional neural network includes but is not limited to: multiple convolutional layers, and the multiple convolutional layers are connected in series. The decoding part of the convolutional neural network includes but is not limited to: multiple convolutional layers and multiple deconvolutional layers, and the multiple convolutional layers and the multiple deconvolutional layers are arranged at intervals and connected in series.

[0122] An optional example of a convolutional neural network in the present disclosure is Figure 5 shown. Figure 5 In the figure, the first rectangle on the left represents the monocular image input to the convolutional neural network, and the first rectangle on the right represents the disparity map output by the convolutional neural network. Each rectangle from the second rectangle on the left to the fifteenth rectangle on the right represents a convolutional layer, and all rectangles from the sixteenth rectangle on the left to the second rectangle on the right represent deconvolutional layers and convolutional layers that are spaced apart from each other, such as the sixteenth rectangle on the left represents a deconvolutional layer, the seventeenth rectangle on the left represents a convolutional layer, the eighteenth rectangle on the left represents a deconvolutional layer, the nineteenth rectangle on the left represents a convolutional layer, and so on, until the second rectangle on the right, and the second rectangle on the right represents a deconvolutional layer.

[0123] In an optional example, the convolutional neural network of the present disclosure can fuse low-level information and high-level information in the convolutional neural network by means of skip connections. For example, the output of at least one convolutional layer in the encoding part is provided to at least one deconvolutional layer in the decoding part by means of skip connections. Optionally, the input of all convolutional layers in the convolutional neural network generally includes: the output of the previous layer (such as a convolutional layer or a deconvolutional layer), and the input of at least one deconvolutional layer in the convolutional neural network (such as a partial deconvolutional layer or all deconvolutional layers) includes: the upsampled result of the output of the previous convolutional layer and the output of the convolutional layer of the encoding part that is skip-connected to the deconvolutional layer. For example, by Figure 5 The solid arrow pointing to the content below the convolutional layer on the right represents the output of the previous convolutional layer. Figure 5 The dotted arrows in the figure represent the upsampling results provided to the deconvolution layer, which are represented by Figure 5 The solid arrow drawn above the convolution layer on the left represents the output of the convolution layer that is jump-connected to the deconvolution layer. The present disclosure does not limit the number of jump connections and the network structure of the convolutional neural network. The present disclosure is beneficial to improving the accuracy of the disparity map generated by the convolutional neural network by fusing the low-level information and high-level information in the convolutional neural network.

[0124] Optionally, the convolutional neural network of the present disclosure is obtained by training with binocular image samples. The training process of the convolutional neural network can be found in the description of the following implementation. No further details are given here.

[0125] In an optional embodiment, the present disclosure may also optimize and adjust the first disparity map of the image to be processed obtained using a convolutional neural network, so as to obtain a more accurate first disparity map. Optionally, in the case where the environmental image is a monocular image, the present disclosure may use the disparity map of the mirror image of the monocular image to optimize and adjust the disparity map of the monocular image, so that the present disclosure can determine multiple obstacle pixel areas in the first disparity map after the disparity adjustment. For ease of description, the mirror image of the monocular image is referred to as the first mirror image, and the disparity map of the first mirror image is referred to as the second disparity map.

[0126] For example, the monocular image in the environment image may be mirrored to obtain a first mirror image, and a disparity map of the first mirror image may be obtained, and then the first disparity map of the monocular image may be disparity-adjusted according to the disparity map of the first mirror image to obtain a first disparity map after disparity adjustment. Subsequently, multiple obstacle pixel areas may be determined in the first disparity map after disparity adjustment.

[0127] A specific example of optimizing and adjusting the first disparity map is as follows:

[0128] Step A: obtaining a second disparity image of a first mirror image of a monocular image, and obtaining a mirror image of the second disparity image.

[0129] Optionally, the first mirror image of the monocular image in the present disclosure may be a mirror image formed by performing a mirror processing (such as left mirror processing or right mirror processing) on ​​the monocular image in the horizontal direction.

[0130] Optionally, the mirror image of the second disparity image in the present disclosure may be a mirror image formed by performing a mirror process (such as left mirror process or right mirror process) on the second disparity image in a horizontal direction. The mirror image of the second disparity image is still a disparity image.

[0131] Optionally, the present disclosure may first perform left mirror processing or right mirror processing on the monocular image (since the left mirror processing result is the same as the right mirror processing result, the present disclosure may perform left mirror processing or right mirror processing on the monocular image), obtain the first mirror image (left mirror image or right mirror image) of the monocular image, and then obtain the disparity map of the first mirror image of the monocular image to obtain the second disparity map; finally, the second disparity map is subjected to left mirror processing or right mirror processing (since the left mirror processing result of the second disparity map is the same as the right mirror processing result, the present disclosure may perform left mirror processing or right mirror processing on the second disparity map), thereby obtaining the mirror image (left mirror image or right mirror image) of the second disparity map. The mirror image of the second disparity map is still a disparity map. For the convenience of description, the mirror image of the second disparity map is referred to as the second mirror image below.

[0132] From the above description, it can be seen that when the present disclosure performs mirror processing on a monocular image, it is not necessary to consider whether the monocular image is mirrored as a left-eye image or a right-eye image. In other words, whether the monocular image is used as a left-eye image or a right-eye image, the present disclosure can perform left-mirror processing or right-mirror processing on the monocular image to obtain a first mirror image. Similarly, when the present disclosure performs mirror processing on the second disparity image, it is also possible not to consider whether the second disparity image should be mirrored to the left or to the right.

[0133] It should be noted that in the process of training a convolutional neural network for generating a disparity map of a monocular image, if the left-eye image sample in the binocular image sample is used as input and provided to the convolutional neural network for training, the convolutional neural network after successful training will use the input monocular image as the left-eye image in testing and actual applications, that is, the monocular image disclosed in the present invention is used as the left-eye image to be processed. If the right-eye image sample in the binocular image sample is used as input and provided to the convolutional neural network for training, the convolutional neural network after successful training will use the input monocular image as the right-eye image in testing and actual applications, that is, the monocular image disclosed in the present invention is used as the right-eye image to be processed.

[0134] Optionally, the present disclosure can also use the above-mentioned convolutional neural network to obtain a second disparity map. For example, the mirror image of the monocular image is input into the convolutional neural network, and the convolutional neural network performs disparity analysis processing on the mirror image of the monocular image, and the convolutional neural network outputs the disparity analysis processing result, so that the present disclosure can obtain the second disparity map according to the output disparity analysis processing result.

[0135] Step B: obtaining a weight distribution map of the first disparity map of the monocular image and a weight distribution map of the second mirror image.

[0136] In an optional example, the weight distribution map of the first disparity map is used to describe the weight values ​​corresponding to each of the multiple disparity values ​​(such as all disparity values) in the first disparity map. The weight distribution map of the first disparity map may include but is not limited to: a first weight distribution map of the first disparity map and a second weight distribution map of the first disparity map.

[0137] Optionally, the first weight distribution map of the first disparity map is a weight distribution map uniformly set for disparity maps of multiple different monocular images, that is, the first weight distribution map of the first disparity map can be oriented to disparity maps of multiple different monocular images, that is, disparity maps of different monocular images use the same first weight distribution map, therefore, the present disclosure may refer to the first weight distribution map of the first disparity map as a global weight distribution map of the first disparity map. The global weight distribution map of the first disparity map is used to describe the global weight values ​​corresponding to each of the multiple disparity values ​​(such as all disparity values) in the first disparity map.

[0138] Optionally, the second weight distribution map of the first disparity map is a weight distribution map set for the disparity map of a single monocular image, that is, the second weight distribution map of the first disparity map is a disparity map for a single monocular image, that is, the disparity maps of different monocular images use different second weight distribution maps, therefore, the present disclosure may refer to the second weight distribution map of the first disparity map as a local weight distribution map of the first disparity map. The local weight distribution map of the first disparity map is used to describe the local weight values ​​corresponding to each of the multiple disparity values ​​(such as all disparity values) in the first disparity map.

[0139] In an optional example, the weight distribution diagram of the second mirror image is used to describe the weight values ​​corresponding to each of the multiple disparity values ​​in the second mirror image. The weight distribution diagram of the second mirror image may include but is not limited to: a first weight distribution diagram of the second mirror image and a second weight distribution diagram of the second mirror image.

[0140] Optionally, the first weight distribution map of the second mirror image is a weight distribution map uniformly set for the mirror images of disparity maps of the mirror images of multiple different monocular images, that is, the first weight distribution map of the second mirror image is a mirror image of the disparity maps of the mirror images of multiple different monocular images, that is, the mirror images of the disparity maps of the mirror images of different monocular images use the same first weight distribution map, therefore, the present disclosure may refer to the first weight distribution map of the second mirror image as the global weight distribution map of the second mirror image. The global weight distribution map of the second mirror image is used to describe the global weight values ​​corresponding to each of the multiple disparity values ​​(such as all disparity values) in the second mirror image.

[0141] Optionally, the second weight distribution map of the second mirror image is a weight distribution map set for the mirror image of the disparity map of the mirror image of a single monocular image, that is, the second weight distribution map of the second mirror image is for the mirror image of the disparity map of the mirror image of a single monocular image, that is, the mirror images of the disparity maps of the mirror images of different monocular images use different second weight distribution maps, therefore, the present disclosure may refer to the second weight distribution map of the second mirror image as a local weight distribution map of the second mirror image. The local weight distribution map of the second mirror image is used to describe the local weight values ​​corresponding to each of the multiple disparity values ​​(such as all disparity values) in the second mirror image.

[0142] In an optional example, the first weight distribution map of the first disparity map includes: at least two left and right areas, and different areas have different weight values. Optionally, the relationship between the weight value of the area on the left and the weight value of the area on the right is usually related to whether the monocular image is used as a left-eye image to be processed or a right-eye image to be processed.

[0143] For example, when a monocular image is used as the left-eye image to be processed, for any two regions in the first weight distribution map of the first disparity map, the weight value of the region on the right is not less than the weight value of the region on the left. Figure 6 for Figure 3 The first weight distribution map of the disparity map shown is divided into five regions, namely Figure 6 Region 1, region 2, region 3, region 4 and region 5 in the first weight distribution map of the first disparity map. The weight value of region 5 is not less than the weight value of region 4, the weight value of region 4 is not less than the weight value of region 3, the weight value of region 3 is not less than the weight value of region 2, and the weight value of region 2 is not less than the weight value of region 1. In addition, any region in the first weight distribution map of the first disparity map may have the same weight value or different weight values. In the case where a region in the first weight distribution map of the first disparity map has different weight values, the weight value of the left part of the region is generally less than or equal to the weight value of the right part of the region. Optionally, Figure 6 The weight value of area 1 in the image can be 0, that is, in the first disparity map, the disparity corresponding to area 1 is completely unreliable; the weight value of area 2 can gradually increase from 0 to 0.5 from the left to the right; the weight value of area 3 is 0.5; the weight value of area 4 can gradually increase from 0.5 to 1 from the left to the right; the weight value of area 5 is 1, that is, in the first disparity map, the disparity corresponding to area 5 is completely credible.

[0144] For another example, when a monocular image is used as the right-eye image to be processed, for any two regions in the first weight distribution map of the first disparity map, the weight value of the region on the left is not less than the weight value of the region on the right. Figure 7 FIG. 4 shows a first weight distribution map of the disparity map of the right eye image to be processed, and the first weight distribution map is divided into five regions, namely Figure 7 Region 1, region 2, region 3, region 4 and region 5 in the first weight distribution map of the first disparity map. The weight value of region 1 is not less than the weight value of region 2, the weight value of region 2 is not less than the weight value of region 3, the weight value of region 3 is not less than the weight value of region 4, and the weight value of region 4 is not less than the weight value of region 5. In addition, any region in the first weight distribution map of the first disparity map may have the same weight value or different weight values. In the case where a region in the first weight distribution map of the first disparity map has different weight values, the weight value of the right part of the region is generally not greater than the weight value of the left part of the region. Optionally, Figure 7The weight value of area 5 can be 0, that is, in the first disparity map, the disparity corresponding to area 5 is completely unreliable; the weight value in area 4 can gradually increase from 0 to 0.5 from the right to the left; the weight value of area 3 is 0.5; the weight value in area 2 can gradually increase from 0.5 to 1 from the right to the left; the weight value of area 1 is 1, that is, in the first disparity map, the disparity corresponding to area 1 is completely credible.

[0145] Optionally, the first weight distribution diagram of the second mirror image includes at least two left and right regions, and different regions have different weight values. Optionally, the relationship between the weight value of the region on the left and the weight value of the region on the right is usually related to whether the monocular image is used as a left-eye image to be processed or a right-eye image to be processed.

[0146] For example, when a monocular image is used as the left-eye image to be processed, for any two areas in the first weight distribution map of the second mirror image, the weight value of the area on the right is not less than the weight value of the area on the left. In addition, any area in the first weight distribution map of the second mirror image can have the same weight value or different weight values. In the case where there are different weight values ​​in an area in the first weight distribution map of the second mirror image, the weight value of the left part in the area is generally not greater than the weight value of the right part in the area.

[0147] For another example, when a monocular image is used as the right-eye image to be processed, for any two areas in the first weight distribution diagram of the second mirror image, the weight value of the area on the left is not less than the weight value of the area on the right. In addition, any area in the first weight distribution diagram of the second mirror image can have the same weight value or different weight values. In the case where there are different weight values ​​in an area in the first weight distribution diagram of the second mirror image, the weight value of the right part in the area is generally not greater than the weight value of the left part in the area.

[0148] Optionally, the method for setting the second weight distribution map of the first disparity map may include the following steps:

[0149] First, left / right mirror processing is performed on the first disparity map to form a mirror disparity map.

[0150] Secondly, according to the disparity value in the mirror disparity map, the weight value in the second weight distribution map of the first disparity map is set.

[0151] Optionally, for a pixel point at any position in a mirror disparity map, if the disparity value of the pixel point at the position satisfies a first predetermined condition, the weight value of the pixel point at the position in the second weight distribution map of the first disparity map is set to a first value, and if the disparity value of the pixel point does not satisfy the first predetermined condition, the weight value of the pixel point at the position in the second weight distribution map of the first disparity map is set to a second value. For example, for a pixel point at any position in a mirror disparity map, if the disparity value of the pixel point at the position is greater than the first reference value corresponding to the pixel point at the position, the weight value of the pixel point at the position in the second weight distribution map of the first disparity map is set to the first value, otherwise, it is set to the second value. The first value in the present disclosure is greater than the second value. For example, the first value is 1 and the second value is 0.

[0152] Optionally, an example of the second weight distribution map of the first disparity map is as follows: Figure 8 shown. Figure 8 The weight values ​​of the white areas in are all 1, indicating that the disparity value at this position is completely credible. Figure 8 The weight value of the black area in is 0, indicating that the disparity value at this position is completely unreliable.

[0153] Optionally, the first reference value corresponding to the pixel point at any position in the present disclosure may be set according to the disparity value of the pixel point at the position in the first disparity map and a constant value greater than 0. For example, the product of the disparity value of the pixel point at the position in the first disparity map and the constant value greater than 0 is used as the first reference value corresponding to the pixel point at the position in the mirror disparity map.

[0154] Optionally, the second weight distribution map of the first disparity map can be expressed using the following formula (1):

[0155]

[0156] In the above formula (1), L l a second weight distribution map representing the first disparity map; represents the disparity value of the pixel at the corresponding position of the mirror disparity map; d l represents the disparity value of the pixel point at the corresponding position in the first disparity map; thresh1 represents a constant value greater than zero, and the value range of thresh1 can be 1.1-1.5, such as thresh1=1.2 or thresh2=1.25.

[0157] In an optional example, the second weight distribution map of the second mirror image may be set in a manner that: according to the disparity value in the first disparity map, the weight value in the second weight distribution map of the second mirror image is set. Optionally, for a pixel point at any position in the second mirror image, if the disparity value of the pixel point at the position in the first disparity map satisfies the second predetermined condition, the weight value of the pixel point at the position in the second weight distribution map of the second mirror image is set to a third value. If the disparity value of the pixel point at the position in the first disparity map does not satisfy the second predetermined condition, the weight value of the pixel point at the position in the second weight distribution map of the second mirror image is set to a fourth value; wherein the third value is greater than the fourth value. For example, for a pixel point at any position in the first disparity map, if the disparity value of the pixel point at the position in the first disparity map is greater than the second reference value corresponding to the pixel point at the position, the weight value of the pixel point at the position in the second weight distribution map of the second mirror image is set to the third value, otherwise, it is set to the fourth value. Optionally, the third value in the present disclosure is greater than the fourth value. For example, the third value is 1 and the fourth value is 0.

[0158] Optionally, the second reference value corresponding to the pixel point in the present disclosure may be set according to the disparity value of the pixel point at the corresponding position in the mirror disparity map and a constant value greater than 0. For example, the first disparity map is first subjected to left / right mirroring to form a mirror disparity map, i.e., a mirror disparity map, and then the product of the disparity value of the pixel point at the corresponding position in the mirror disparity map and the constant value greater than 0 is used as the second reference value corresponding to the pixel point at the corresponding position in the first disparity map.

[0159] Optional, based on Figure 2 An example of the second mirror image formed by the environment image is as follows Fig. 9 shown. Fig. 9 An example of a second weight distribution diagram of the second mirror image is shown in Fig.10 shown. Fig.10 The weight values ​​of the white areas in are all 1, indicating that the disparity value at this position is completely credible. Fig.10 The weight value of the black area in is 0, indicating that the disparity value at this position is completely unreliable.

[0160] Optionally, the second weight distribution graph of the second mirror image can be expressed using the following formula (2):

[0161]

[0162] In the above formula (2), L l ′ represents the second weight distribution graph of the second mirror image; represents the disparity value of the pixel at the corresponding position of the mirror disparity map; d lrepresents the disparity value of the pixel point at the corresponding position in the first disparity map; thresh2 represents a constant value greater than zero, and the value range of thresh2 can be 1.1-1.5, such as thresh2=1.2 or thresh2=1.25.

[0163] Step C: optimizing and adjusting the first disparity map of the monocular image according to the weight distribution map of the first disparity map of the monocular image and the weight distribution map of the second mirror image. The disparity map after optimization and adjustment is the first disparity map of the monocular image finally obtained.

[0164] In an optional example, the present disclosure can use the first weight distribution map and the second weight distribution map of the first disparity map to adjust multiple disparity values ​​in the first disparity map to obtain an adjusted first disparity map; use the first weight distribution map and the second weight distribution map of the second mirror image to adjust multiple disparity values ​​in the second mirror image to obtain an adjusted second mirror image; then, merge the adjusted first disparity map and the adjusted second mirror image to obtain an optimized and adjusted first disparity map of the monocular image.

[0165] Optionally, an example of obtaining the first disparity map of the optimized and adjusted monocular image is as follows:

[0166] First, the first weight distribution map of the first disparity map and the second weight distribution map of the first disparity map are combined to obtain a third weight distribution map. The third weight distribution map can be expressed by the following formula (3):

[0167] W l =M l +L l ·0.5 Formula (3)

[0168] In formula (3), W l represents the third weight distribution diagram; M l A first weight distribution map representing a first disparity map; L l The second weight distribution map representing the first disparity map, wherein 0.5 may also be transformed into other constant values.

[0169] Secondly, the first weight distribution map of the second mirror image and the second weight distribution map of the second mirror image are combined to obtain a fourth weight distribution map. The fourth weight distribution map can be expressed by the following formula (4):

[0170] W l '=M l '+L l '·0.5 Formula (4)

[0171] In formula (4), W l' represents the fourth weight distribution diagram, M l ' represents the first weight distribution diagram of the second mirror image; L l ' represents the second weight distribution diagram of the second mirror image, wherein 0.5 can also be transformed into other constant values.

[0172] Again, multiple disparity values ​​in the first disparity map are adjusted according to the third weight distribution map to obtain an adjusted first disparity map. For example, for the disparity value of a pixel at any position in the first disparity map, the disparity value of the pixel at the position is replaced by: the product of the disparity value of the pixel at the position and the weight value of the pixel at the corresponding position in the third weight distribution map. After the above replacement process is performed on all pixels in the first disparity map, the adjusted first disparity map is obtained.

[0173] And, adjusting multiple disparity values ​​in the second mirror image according to the fourth weight distribution map to obtain an adjusted second mirror image. For example, for the disparity value of a pixel point at any position in the second mirror image, the disparity value of the pixel point at the position is replaced by: the product of the disparity value of the pixel point at the position and the weight value of the pixel point at the corresponding position in the fourth weight distribution map. After performing the above replacement process on all the pixels in the second mirror image, the adjusted second mirror image is obtained.

[0174] Finally, the adjusted first disparity map and the adjusted second mirror image are combined to finally obtain the first disparity map of the monocular image. The first disparity map of the monocular image finally obtained can be expressed by the following formula (5):

[0175]

[0176] In formula (5), d final The first disparity map of the monocular image finally obtained (such as Fig.11 (as shown in the first figure on the right in the figure); W l Represents the third weight distribution diagram (such as Fig.11 (as shown in the first picture on the upper left of the figure); W l ' represents the fourth weight distribution diagram (such as Fig.11 (as shown in the first figure on the lower left); d l Represents the first disparity map (such as Fig.11 As shown in the second picture from the upper left in the figure); Represents the second mirror image (such as Fig.11 (as shown in the second figure from the lower left in the figure).

[0177] It should be noted that the present disclosure does not limit the execution order of the two steps of merging the first weight distribution map and the second weight distribution map. For example, the two merging steps can be executed simultaneously or sequentially. In addition, the present disclosure does not limit the execution order of adjusting the disparity value in the first disparity map and adjusting the disparity value in the second mirror image. For example, the two adjustment steps can be performed simultaneously or sequentially.

[0178] When a monocular image is used as a left-eye image to be processed, there are usually phenomena such as missing left-side disparity and occlusion of the left edge of the object, which will cause the disparity value of the corresponding area in the first disparity map of the monocular image to be inaccurate. Similarly, when a monocular image is used as a right-eye image to be processed, there are usually phenomena such as missing right-side disparity and occlusion of the right edge of the object, which will cause the disparity value of the corresponding area in the first disparity map of the monocular image to be inaccurate. The present disclosure performs mirror processing on the monocular image, and mirror processing on the disparity map of the first mirror image (i.e., the second disparity map), and then uses the mirrored disparity map (i.e., the second mirror image) to optimize and adjust the first disparity map of the monocular image, which is beneficial to reduce the phenomenon of inaccurate disparity values ​​of the corresponding area in the first disparity map of the monocular image, thereby helping to improve the accuracy of obstacle detection.

[0179] In an optional example, in an application scenario where the environmental image is a binocular image to be processed, the method of obtaining the first disparity map of the binocular image to be processed disclosed herein includes but is not limited to: obtaining the first disparity map of the binocular image to be processed by stereo matching. For example, the first disparity map of the binocular image to be processed is obtained by using a stereo matching algorithm such as a BM (Block Matching) algorithm, a SGBM (Semi-Global BlockMatching) algorithm, or a GC (Graph Cuts) algorithm. For another example, a convolutional neural network for obtaining a disparity map of a binocular image is used to perform disparity processing on the binocular image to be processed, thereby obtaining the first disparity map of the binocular image to be processed.

[0180] S110 . Determine a plurality of obstacle pixel regions in a first disparity map of an environment image.

[0181] Exemplarily, the obstacle pixel region may be a pixel region containing at least two pixels in a continuous position in the first disparity map. In one embodiment, the obstacle pixel region may be an obstacle pixel column region. For example, the obstacle pixel column region in the present disclosure is a strip region, the width of the strip region is at least one column of pixels, and the height of the strip region is at least two rows of pixels. Since the strip region can be used as a basic unit of an obstacle, the present disclosure refers to the strip region as an obstacle pixel column region.

[0182] In an optional example, the present disclosure may first perform edge detection on the first disparity map of the environment image obtained in the above step to obtain obstacle edge information; then, determine the obstacle area in the first disparity map of the environment image; finally, determine multiple obstacle pixel columnar areas in the obstacle area based on the obstacle edge information. The present disclosure helps avoid the phenomenon of forming obstacle pixel columnar areas in areas with low attention value by dividing the obstacle area, and helps improve the convenience of forming obstacle pixel columnar areas.

[0183] Different obstacles in the actual space will cause different parallaxes due to their different distances from the camera device, thus forming parallax edges of the obstacles. The present disclosure can separate obstacles in the disparity map by detecting obstacle edge information, so that the present disclosure can conveniently form obstacle pixel columnar areas by searching for obstacle edge information, which is conducive to improving the convenience of forming obstacle pixel columnar areas.

[0184] In an optional example, the present invention discloses a method for obtaining obstacle edge information in a first disparity map of an environment image, including but not limited to: using a convolutional neural network for edge extraction to obtain obstacle edge information in the first disparity map of an environment image; and using an edge detection algorithm to obtain obstacle edge information in the first disparity map of an environment image.

[0185] Optionally, the present disclosure uses an edge detection algorithm to obtain obstacle edge information in the first disparity map of the environment image. Fig.12 shown.

[0186] Fig.12 In step 1, a histogram equalization process is performed on the first disparity map of the environment image. The first disparity map of the environment image is as follows: Fig.12 The first disparity map can be finally obtained by using the above step 100. Figure 2 The result of histogram equalization is shown in Fig.12 As shown in the second picture from the upper left.

[0187] Step 2: Perform mean filtering on the result of histogram equalization. The result after filtering is as follows: Fig.12 As shown in the third picture from the upper left.

[0188] The above steps 1 and 2 are preprocessing of the first disparity map of the environment image. Steps 1 and 2 are only an example of preprocessing the first disparity map of the environment image. The present disclosure does not limit the specific implementation of the preprocessing.

[0189] Step 3: Use edge detection algorithm to perform edge detection on the result after filtering to obtain edge information. The edge information obtained in this step is as follows: Fig.12 The edge detection algorithm in the present disclosure includes but is not limited to: Canny edge detection algorithm, Sobel edge detection algorithm or Laplacian edge detection algorithm.

[0190] Step 4: Perform morphological dilation operation on the edge information obtained. The result of the dilation operation is as follows: Fig.12 This step belongs to a post-processing method for the detection result of the edge detection algorithm. The present disclosure does not limit the specific implementation method of the post-processing.

[0191] Step 5: Perform a reverse operation on the result of the dilation operation to obtain an edge mask (Mask) of the first disparity image of the environment image. The edge mask of the first disparity image of the environment image is as follows: Fig.12 As shown in the lower left corner of the figure.

[0192] Step 6: Perform an AND operation on the edge mask of the first disparity map of the environment image and the first disparity map of the environment image to obtain obstacle edge information in the first disparity map of the environment image. Fig.12 The figure on the right shows the obstacle edge information in the first disparity map of the environment image. For example, the disparity value at the obstacle edge position in the first disparity map of the environment image is set to 0. Fig.12 In the figure, it appears as a black edge line.

[0193] In an optional example, the present disclosure determines an obstacle area in a first disparity map of an environment image, including the following steps:

[0194] Step a: statistically process the disparity values ​​of each row of pixels in the first disparity map to obtain statistical information of the disparity values ​​of each row of pixels, and determine a statistical disparity map based on the statistical information of the disparity values ​​of each row of pixels.

[0195] Optionally, the present disclosure may perform horizontal statistics (row statistics) on the first disparity map of the environmental image to obtain a V disparity map, which can be used as a statistical disparity map. That is, for each row of the first disparity map of the environmental image, the number of each disparity value in the row is counted, and the statistical result is set on the corresponding column of the V disparity map. The width of the V disparity map is related to the value range of the disparity value. For example, if the value range of the disparity value is 0-254, the width of the V disparity map is 255. The height of the V disparity map is the same as the height of the first disparity map of the environmental image, that is, the number of rows contained in both is the same.

[0196] Optional, for Figure 4For the first disparity map of the environment image shown in FIG. 1 , the statistical disparity map formed by the present disclosure is as follows: Fig.13 shown. Fig.13 In the figure, the top row represents the disparity values ​​0 to 5; the value of the second row and the first column is 1, indicating Figure 4 The number of disparity values ​​in the first row of 0 is 1; the value in the second row and second column is 6, indicating Figure 4 The number of disparity values ​​1 in the first row is 6; the value in the fifth row and sixth column is 5, indicating Figure 4 The number of disparity values ​​5 in the 5th row of is 5. Fig.13 The other values ​​in are explained one by one.

[0197] Optional, for Fig.14 For the first disparity map of the environment image shown in the left figure in FIG, the first disparity map of the environment image is processed, and the obtained V disparity map is as follows Fig.14 As shown in the right figure.

[0198] Step b: performing a first straight line fitting process on the statistical disparity map (also referred to as a V disparity map in the present disclosure), and determining a ground area and a non-ground area according to a result of the first straight line fitting process.

[0199] First, the present disclosure may pre-process the V disparity map. The pre-processing of the V disparity map may include but is not limited to: removing noise, etc. For example, threshold filtering is performed on the V disparity map to filter out noise in the V disparity map. Fig.15 In the case shown in the first figure on the left, the V disparity map after filtering out the noise is as follows Fig.15 As shown in the second picture on the left.

[0200] Secondly, the present disclosure performs a first straight line fitting (fitline) on the V disparity map after noise removal, thereby obtaining a first straight line equation v=Ad+B, wherein v represents the row coordinate in the V disparity map, and d represents the disparity value.

[0201] For example, Fig.13 The slash in represents the first fitted straight line equation. For another example, Fig.15 The white oblique line in the first figure on the right side of the diagram represents the first straight line equation fitted. The first straight line fitting method includes but is not limited to: RANSAC straight line fitting method.

[0202] Optionally, the first straight line equation obtained by the above fitting can express the relationship between the disparity value of the ground area and the row coordinates of the V disparity map. That is, for any row in the V disparity map, when v is determined, the disparity value d of the ground area should be a determined value. The disparity value of the ground area can be expressed in the form of the following formula (6):

[0203]

[0204] In formula (6), d road represents the parallax of the ground area; A and B are known values, such as those obtained by the first straight line fitting.

[0205] Again, the present disclosure can use formula (6) to segment the first disparity map of the environment image to obtain the ground area I road and non-ground area I notroad .

[0206] Optionally, the present disclosure may use the following formula (7) to determine the ground area and the non-ground area:

[0207]

[0208] In the above formula (7), I(*) represents a pixel set. If the disparity value of a pixel in the first disparity map of the environment image satisfies |dd road |≤thresh3, then the pixel belongs to the ground area I road ; If the disparity value of a pixel in the first disparity map of the environment image satisfies |dd road |>thresh3, the pixel belongs to the non-ground area; thresh3 represents a threshold value, which is a known value. The size of the threshold value can be set according to the actual situation.

[0209] Optional, Ground Area I road You can Fig.16 As shown in the upper right figure of the figure. Non-ground area I notroad You can Fig.16 As shown in the lower right figure.

[0210] The present disclosure helps to remove the influence of noise in the first disparity map of the environment image on region judgment by setting a threshold, thereby helping to more accurately determine the ground area and the non-ground area.

[0211] Finally, the obstacle area is determined based on the non-ground area.

[0212] Optionally, the non-ground area I in the present disclosure notroad It may include: a first area I above the ground high and a second area I below the ground low At least one of the above. The present disclosure may include the non-ground area I notroad The area above the ground and whose height above the ground is less than a predetermined height value is regarded as an obstacle area. low It may be a pit, a ditch or a valley, so the present invention may include a non-ground area Inotroad The area below the ground and whose height below the ground is less than a predetermined height value is regarded as an obstacle area.

[0213] The first area I above the ground in the present disclosure high and a second area I below the ground low It can be expressed by the following formula (8):

[0214]

[0215] In the above formula (8), I notroad (*) indicates a pixel set, if the disparity value of a pixel in the first disparity map of the environment image satisfies dd road >thresh4, the pixel belongs to the first region I above the ground high ; If the disparity value of a pixel in the first disparity map of the environment image satisfies d road -d>thresh4, then the pixel belongs to the second area I below the ground low ; thresh4 represents a threshold value, which is a known value. The value of the threshold value can be set according to the actual situation.

[0216] Optionally, in a first area I above the ground high In the above description, obstacles that do not need to be paid attention to are often included, such as traffic lights and overpasses. Since they do not affect the driving of the vehicle, these target objects are obstacles that do not need to be paid attention to for the vehicle. These obstacles that do not need to be paid attention to are often located at a higher position and will not affect the driving of the vehicle or the walking of pedestrians. high In the example, the area at a higher position is removed, for example, the area whose height above the ground is greater than or equal to the first predetermined height value is removed, thereby forming the obstacle area I obstacle .

[0217] Optionally, the present disclosure may perform a second straight line fitting process according to the V disparity map, and determine the area in the non-ground area that needs to be removed and belongs to a higher position (i.e., the area whose height above the ground is greater than or equal to the first predetermined height value) according to the result of the second straight line fitting process, thereby obtaining the obstacle area I in the non-ground area. obstacle The second straight line fitting method includes but is not limited to: a RANSAC straight line fitting method.

[0218] Optionally, when there is a second area below the ground in the non-ground area, a second target area whose height below the ground is greater than a second predetermined height value in the second area is determined, and the second target area is an obstacle area.

[0219] The present disclosure performs a second straight line fitting on the V disparity map, and the obtained second straight line equation can be expressed as v=Cd+D. Where v represents the row coordinate in the V disparity map, and d represents the disparity value. Through derivation and calculation, it can be known that C and D can be expressed as: Therefore, the second straight line equation of the present disclosure can be expressed as: H is a known constant value, and H can be set according to actual needs. For example, in the intelligent control technology of the vehicle, H can be set to 2.5 meters.

[0220] Optional, Fig.18 The middle image in the figure contains two white oblique lines, one above the other. The upper white oblique line represents the equation of the second fitted line.

[0221] Optionally, the second straight line equation obtained by the above fitting can represent the relationship between the disparity value of the obstacle area and the row coordinates of the disparity map V. That is, for any row in the disparity map V, when v is determined, the disparity value d of the obstacle area should be a determined value.

[0222] Optionally, the present disclosure may include a first area I above the ground. high Divided into the form expressed by the following formula (9):

[0223]

[0224] In the above formula (9), I high (*) represents a pixel set, if the disparity value d of a pixel in the first disparity map of the environment image satisfies d<d H , then the pixel belongs to area I which is above the ground but below the height H above the ground ＜H , the present disclosure can be ＜H As obstacle area I obstacl ; If the disparity value d of a pixel in the first disparity map of the environment image satisfies d>d H , then the pixel belongs to area I which is above the ground and higher than the height H above the ground ＞H ;d H Indicates the disparity value of the pixel at height H above the ground; I ＞H You can Fig.18 As shown in the upper right figure. ＜H You can Fig.18 As shown in the lower right figure.

[0225] In the above formula (9):

[0226] In an optional example, the present disclosure is in the obstacle area I obstacleThe method of determining the pixel columnar area according to the obstacle edge information can be: first, the disparity values ​​of the pixels in the non-obstacle area of ​​the first disparity map and the disparity values ​​of the pixels at the obstacle edge information are set to predetermined values. Secondly, the N pixels in the column direction of the first disparity map are used as traversal units, and the disparity values ​​of the N pixels on each row are traversed from the set row of the first disparity map to determine the target row where the disparity value of the pixel point jumps between the predetermined value and the non-predetermined value; finally, the N pixels in the column direction are used as the column width, and the determined target row is used as the boundary of the obstacle pixel columnar area in the row direction to determine the obstacle pixel columnar area in the obstacle area.

[0227] For example, the present disclosure is in obstacle area I obstacle The method of determining the pixel columnar area according to the obstacle edge information can be: first, according to the detected obstacle edge information, the disparity values ​​at the obstacle edge position in the disparity map are all set to predetermined values ​​(such as 0), and the disparity values ​​in the area other than the obstacle area in the disparity map are also set to predetermined values ​​(such as 0); then, according to the predetermined column width (at least one column of pixel width, such as 6 columns of pixel width, etc.), search from the bottom of the disparity map upward, and when the disparity value of any column of pixels in the predetermined column width is searched to jump from the predetermined value to the non-predetermined value, the position (row of the disparity map) is determined to be the bottom of the pixel columnar area, and the pixel columnar area begins to be formed, that is, the pixel columnar area begins to be extended upward, for example, the disparity map continues to be searched upward for the jump from the non-predetermined value to the predetermined value, and when the disparity value of any column of pixels in the predetermined column width is searched to jump from the non-predetermined value to the predetermined value, the upward extension of the pixel columnar area is stopped, and the position (row of the disparity map) is determined to be the top of the pixel columnar area, thereby forming an obstacle pixel columnar area.

[0228] It should be noted that the present disclosure can start the process of determining the obstacle pixel columnar area from the lower left corner of the disparity map until the lower right corner of the disparity map. For example, the process of determining the obstacle pixel columnar area is performed from the 6 columns on the leftmost side of the disparity map, and then the process of determining the obstacle pixel columnar area is performed again from the 7th to 12th columns on the leftmost side of the disparity map until the rightmost column of the disparity map. The present disclosure can also start the process of determining the obstacle pixel columnar area from the lower right corner of the disparity map until the lower left corner of the disparity map. In addition, it is also completely feasible to expand from the middle position of the lower end of the disparity map to both sides to form an obstacle pixel columnar area.

[0229] In an optional example, the present disclosure is in the obstacle area I obstacleThe method of forming a pixel columnar region according to the obstacle edge information can be: first, according to the detected obstacle edge information, the disparity values ​​at the obstacle edge position in the disparity map are all set to predetermined values ​​(such as 0), and the disparity values ​​in the area other than the obstacle area in the disparity map are also set to predetermined values ​​(such as 0); then, according to the predetermined column width (at least one column of pixel width, such as 6 columns of pixel width, etc.), search from the top of the disparity map downward, and when the disparity value of any column of pixels in the predetermined column width is searched to jump from the predetermined value to the non-predetermined value, the position (row of the disparity map) is determined to be the top of the pixel columnar region, and the pixel columnar region is started to be formed, that is, the pixel columnar region is started to be extended downward, for example, the disparity map is continued to be searched upward for the jump from the non-predetermined value to the predetermined value, and when the disparity value of any column of pixels in the predetermined column width is searched to jump from the non-predetermined value to the predetermined value, the downward extension of the pixel columnar region is stopped, and the position (row of the disparity map) is determined to be the bottom of the pixel columnar region, thereby forming an obstacle pixel columnar region.

[0230] It should be noted that the present disclosure can start the process of determining the obstacle pixel columnar area from the upper left corner of the disparity map until the upper right corner of the disparity map. For example, the process of determining the obstacle pixel columnar area is performed from the 6 columns at the top of the leftmost side of the disparity map, and then the process of determining the obstacle pixel columnar area is performed again from the 7th to 12th columns at the top of the leftmost side of the disparity map until the rightmost column of the disparity map. The present disclosure can also start the process of determining the obstacle pixel columnar area from the upper right corner of the disparity map until the upper left corner of the disparity map. In addition, it is also completely feasible to expand from the middle position of the uppermost end of the disparity map to both sides to form an obstacle pixel columnar area.

[0231] As can be seen from the above description, the obstacle pixel column region in the present disclosure is a strip region, the width of the strip region is at least one column of pixels, and the height of the strip region is at least two rows of pixels. Since the strip region is used as the basic unit of the obstacle, the present disclosure refers to the strip region as the obstacle pixel column region.

[0232] Optionally, the present disclosure is directed to Figure 2 An example of the obstacle pixel column area formed by the environment image is shown in Fig.19 As shown in the picture on the right. Fig.19 The width of each obstacle pixel column area in the right figure is 6 columns of pixels. The width of the obstacle pixel column area can be set according to actual needs. The larger the width of the obstacle pixel column area is set, the rougher the obstacle pixel column area is formed, and the time it takes to form the obstacle pixel column area will also be shorter.

[0233] In an optional example, after the obstacle pixel column area is formed, the attribute information of the obstacle pixel column area should be determined, and the attribute information of the obstacle pixel column area includes but is not limited to: spatial position information of the obstacle pixel column area, bottom information bottom of the obstacle pixel column area, disparity value disp of the obstacle pixel column area, top information top of the obstacle pixel column area, and column information col of the obstacle pixel column area.

[0234] Optionally, the spatial position information of the obstacle pixel column area may include: the coordinates of the obstacle pixel column area on the horizontal coordinate axis (X coordinate axis), the coordinates of the obstacle pixel column area on the depth coordinate axis (Z coordinate axis), the coordinates of the highest point of the obstacle pixel column area on the vertical coordinate axis (Y coordinate axis), and the coordinates of the lowest point of the obstacle pixel column area on the vertical coordinate axis (Y coordinate axis). In other words, the spatial position information of the obstacle pixel column area includes: the X coordinate, Z coordinate, maximum Y coordinate, and minimum Y coordinate of the obstacle pixel column area. An example of the X coordinate axis, Y coordinate axis, and Z coordinate axis is as follows: Fig.17 shown.

[0235] Optionally, the bottom information of the obstacle pixel column area may be the row number of the bottom end of the obstacle pixel column area. When the predetermined value is 0, the disparity value of the obstacle pixel column area may be: when the disparity value jumps from zero to non-zero, the disparity value of the pixel at the non-zero position; the top information of the obstacle pixel column area may be the row number of the pixel at the zero position when the disparity value jumps from non-zero to zero. The column information of the obstacle pixel column area may be the column number of any column among all the columns included in the device pixel, for example, the column number of a column located in the middle of the pixel column area.

[0236] Optionally, for each obstacle pixel column area, the present disclosure uses the following formula (10) to calculate the spatial position information of the obstacle pixel column area, that is, the X coordinate, Z coordinate, maximum Y coordinate and minimum Y coordinate of the obstacle pixel column area:

[0237]

[0238] In the above formula (10), b represents the distance between the binocular cameras; f represents the focal length of the camera; Disp represents the disparity value of the obstacle pixel column area; Col represents the column information of the obstacle pixel column area; c x Indicates the X coordinate value of the camera's principal point.

[0239] Optionally, the Y coordinate of each pixel in the obstacle pixel column area can be expressed using the following formula (11):

[0240]

[0241] In the above formula (11), Y i Represents the Y coordinate of the i-th pixel in the obstacle pixel column area; row i represents the row number of the i-th pixel in the obstacle pixel column area; c y represents the Y coordinate value of the principal point of the camera device; Z represents the Z coordinate of the obstacle pixel column area; f represents the focal length of the camera device.

[0242] After obtaining the Y coordinates of all pixels in an obstacle pixel column area, the maximum Y coordinate and the minimum Y coordinate can be obtained. The maximum Y coordinate and the minimum Y coordinate can be expressed as shown in the following formula (12):

[0243] Y min =min(Y i )

[0244] Y max =max(Y i ) Formula (12)

[0245] In the above formula (12), Y min Indicates the minimum Y coordinate of the obstacle pixel column area; Y max Indicates the maximum Y coordinate of the obstacle pixel column area; min(Y i ) means to calculate all Y i The minimum value in max(Y i ) means to calculate all Y i The maximum value in .

[0246] S120: Perform clustering processing on multiple obstacle pixel regions to obtain at least one cluster.

[0247] In an optional example, the present disclosure may cluster multiple obstacle pixel columnar regions to obtain at least one cluster. The present disclosure may cluster all obstacle pixel columnar regions according to spatial position information of the obstacle pixel columnar regions, and one cluster corresponds to one obstacle instance. The present disclosure may cluster each obstacle pixel columnar region using a corresponding clustering algorithm.

[0248] Optionally, before clustering the plurality of pixel columnar regions, the X-coordinates and Z-coordinates of the pixel columnar regions of the obstacle may be normalized (ie, normalized).

[0249] For example, the present disclosure may use a min-max normalization processing method to map the X coordinate and Z coordinate of the obstacle pixel column area so that the X coordinate and Z coordinate of the obstacle pixel column area are mapped to a value range of [0-1]. An example of this normalization processing method is shown in the following formula (13):

[0250]

[0251] In the above formula (13), X * represents the normalized X coordinate; Z * represents the normalized Z coordinate; X represents the X coordinate of the obstacle pixel column area; Z represents the Z coordinate of the obstacle pixel column area; X min Indicates the minimum value of the X coordinates of all obstacle pixel columnar areas; X max Indicates the maximum value of the X coordinate of all obstacle pixel columnar areas; Z min Represents the minimum value of the Z coordinates of all obstacle pixel columnar areas; Z max Indicates the maximum value of the Z coordinates of all obstacle pixel columnar areas.

[0252] For another example, the present disclosure may also use a Z-score normalization processing method to normalize the X coordinates and Z coordinates of the obstacle pixel columnar area. An example of this normalization processing method is shown in the following formula (14):

[0253]

[0254] In the above formula (14), X * represents the normalized X coordinate; Z * represents the normalized Z coordinate; X represents the X coordinate of the obstacle pixel column area; Z represents the Z coordinate of the obstacle pixel column area; μ X represents the mean value calculated for the X coordinates of all obstacle pixel columnar areas; σ X represents the standard deviation calculated for the X coordinates of all obstacle pixel columnar areas; μ Z represents the mean value calculated for the Z coordinates of all obstacle pixel columnar areas; σ Z The standard deviation of the Z coordinates of all obstacle pixel columnar regions after processing is shown in FIG. * and Z * All of them conform to the standard normal distribution, that is, the mean is 0 and the standard deviation is 1.

[0255] Optionally, the present disclosure may use a density clustering (DBSCAN) algorithm to cluster the obstacle pixel columnar regions according to the spatial position information of all obstacle pixel columnar regions after normalization, thereby forming at least one cluster, each of which is an obstacle example. The present disclosure does not limit the clustering algorithm. An example of a clustering result is as follows: Fig. 20 As shown in the right figure.

[0256] S130: Determine an obstacle detection result according to obstacle pixel regions belonging to the same cluster.

[0257] Exemplarily, the obstacle detection result may include, for example but not limited to, at least one of an obstacle detection frame and spatial position information of the obstacle.

[0258] In an optional example, the present disclosure may determine the obstacle detection box (Bounding-Box) in the environment image based on the spatial position information of the pixel columnar regions belonging to the same cluster. For example, for a cluster, the present disclosure may calculate the maximum column coordinate u of all obstacle pixel columnar regions in the cluster in the environment image. max and the minimum column coordinate u min , and calculate the maximum bottom (i.e. v max ) and the smallest top (i.e. v min ) (Note: Assume that the origin of the image coordinate system is located at the upper left corner of the image). The coordinates of the obstacle detection frame obtained in the present disclosure in the environment image can be expressed as (u min , v min ,u max , v max ).

[0259] Optionally, an example of an obstacle detection frame determined by the present disclosure is as follows: Fig.21 As shown in the picture on the right. Fig.21 The multiple rectangular frames in the right figure are all obstacle detection frames obtained by the present invention.

[0260] The present disclosure obtains obstacles by clustering multiple obstacle pixel columnar areas. It is not necessary to predefine the obstacles to be detected, and does not use predefined information such as the texture, color, shape, and category of the obstacles. Obstacles can be detected directly based on clustering obstacle areas. Moreover, the detected obstacles are not limited to certain predefined obstacles, and various obstacles in the surrounding space environment that may hinder the movement of smart devices can be detected, thereby realizing universal obstacle detection.

[0261] In an optional example, the present disclosure may also determine the spatial position information of the obstacle based on the spatial position information of multiple obstacle pixel columnar regions belonging to the same cluster. The spatial position information of the obstacle may include, but is not limited to: the coordinates of the obstacle on the horizontal coordinate axis (X coordinate axis), the coordinates of the obstacle on the depth coordinate axis (Z coordinate axis), and the height of the obstacle in the vertical direction (i.e., the height of the obstacle).

[0262] Optionally, the present disclosure can first determine the distance between multiple obstacle pixel column areas in a cluster and a camera device that generates an environmental image based on the spatial position information of multiple obstacle pixel column areas belonging to the same cluster, and then determine the spatial position information of the obstacle based on the spatial position information of the obstacle pixel column area that is closest to the cluster.

[0263] Optionally, the present disclosure may use the following formula (15) to calculate the distance between multiple obstacle pixel columnar regions in a cluster and the camera device, and select the minimum distance:

[0264]

[0265] In the above formula (15), d min Indicates the minimum distance; X i represents the X coordinate of the pixel column area of ​​the ith obstacle in a cluster; Z i Represents the Z coordinate of the pixel columnar area of ​​the i-th obstacle in a cluster.

[0266] After the minimum distance is determined, the X coordinate and Z coordinate of the obstacle pixel column area with the minimum distance can be used as the spatial position information of the obstacle, as shown in the following formula (16):

[0267] O X =X close

[0268] O Z =Z close Formula (16)

[0269] In the above formula (16), O X Represents the coordinate of the obstacle on the horizontal coordinate axis, that is, the X coordinate of the obstacle; Z Indicates the coordinate of the obstacle on the depth coordinate axis (X coordinate axis), that is, the Z coordinate of the obstacle; close represents the X coordinate of the obstacle pixel column area with the minimum distance calculated above; Z close Indicates the Z coordinate of the obstacle pixel column area with the minimum distance calculated above.

[0270] Optionally, the present disclosure may use the following formula (17) to calculate the height of the obstacle:

[0271] O H =Y max -Y min Formula (17)

[0272] In the above formula (17), O H Indicates the height of the obstacle; Y max represents the maximum Y coordinate of all obstacle pixel columnar areas in a cluster; min Represents the minimum Y coordinate of all obstacle pixel columnar areas in a cluster.

[0273] The present invention discloses a process of training a convolutional neural network in one embodiment, such as Fig. 22 shown.

[0274] S2200: Input one of the binocular image samples (such as the left / right eye) into the convolutional neural network to be trained.

[0275] Optionally, the image samples input into the convolutional neural network of the present disclosure may always be left-eye image samples of binocular image samples, or may always be right-eye image samples of binocular image samples. In the case where the image samples input into the convolutional neural network are always left-eye image samples of binocular image samples, the convolutional neural network after successful training will use the input environmental image as the left-eye image to be processed in the test or actual application scenario. In the case where the image samples input into the convolutional neural network are always right-eye image samples of binocular image samples, the convolutional neural network after successful training will use the input environmental image as the right-eye image to be processed in the test or actual application scenario.

[0276] S2210, performing disparity analysis processing via a convolutional neural network, and obtaining a disparity map of a left-eye image sample and a disparity map of a right-eye image sample based on an output of the convolutional neural network.

[0277] S2220 , reconstructing the right-eye image according to the disparity map of the left-eye image samples and the right-eye image samples.

[0278] Optionally, the method of reconstructing the right-eye image disclosed herein includes but is not limited to: performing reprojection calculation on the disparity maps of the left-eye image samples and the right-eye image samples, so as to obtain the reconstructed right-eye image.

[0279] S2230 , reconstructing the left-eye image according to the disparity map of the right-eye image samples and the left-eye image samples.

[0280] Optionally, the method of reconstructing the left-eye image disclosed herein includes but is not limited to: performing reprojection calculation on the disparity maps of the right-eye image samples and the left-eye image samples, so as to obtain the reconstructed left-eye image.

[0281] S2240, adjusting the network parameters of the convolutional neural network according to the difference between the reconstructed left-eye image and the left-eye image sample, and the difference between the reconstructed right-eye image and the right-eye image sample.

[0282] Optionally, when determining the difference, the loss function used by the present disclosure includes but is not limited to: L1 loss function, smooth loss function, and lr-Consistency loss function, etc. In addition, when the present disclosure back-propagates the calculated loss to adjust the network parameters of the convolutional neural network (such as the weights of the convolution kernel), the loss can be back-propagated based on the gradient calculated by the chain derivation of the convolutional neural network, thereby facilitating improving the training efficiency of the convolutional neural network.

[0283] In an optional example, when the training of the convolutional neural network reaches a predetermined iteration condition, the training process ends. The predetermined iteration condition in the present disclosure may include: the difference between the left eye image reconstructed based on the disparity map output by the convolutional neural network and the left eye image sample, and the difference between the right eye image reconstructed based on the disparity map output by the convolutional neural network and the right eye image sample, meet the predetermined difference requirement. In the case where the difference meets the requirement, the convolutional neural network is successfully trained this time. The predetermined iteration condition in the present disclosure may also include: the convolutional neural network is trained, and the number of binocular image samples used reaches the predetermined number requirement, etc. When the number of binocular image samples used reaches the predetermined number requirement, however, the difference between the left eye image reconstructed based on the disparity map output by the convolutional neural network and the left eye image sample, and the difference between the right eye image reconstructed based on the disparity map output by the convolutional neural network and the right eye image sample, do not meet the predetermined difference requirement, the convolutional neural network is not successfully trained this time.

[0284] Fig.23 The figure is a flow chart of an embodiment of the intelligent driving control method disclosed in the present invention. The intelligent driving control method disclosed in the present invention can be applied to, but not limited to, an autonomous driving (such as fully unassisted autonomous driving) environment or an assisted driving environment.

[0285] S2300: Acquire an environment image of the intelligent device during movement through an image acquisition device provided on the intelligent driving device. The image acquisition device includes but is not limited to: an RGB-based camera device, etc.

[0286] S2310: Perform obstacle detection on the acquired environment image and determine the obstacle detection result. Figure 1 The description will not be described in detail here.

[0287] S2320: Generate and output control instructions according to the obstacle detection result.

[0288] Optionally, the control instructions generated by the present invention include but are not limited to: speed maintaining control instructions, speed adjustment control instructions (such as deceleration instructions, acceleration instructions, etc.), direction maintaining control instructions, direction adjustment control instructions (such as left turn instructions, right turn instructions, merging into the left lane instructions, or merging into the right lane instructions, etc.), horn instructions, early warning control instructions or driving mode switching control instructions (such as switching to automatic cruise driving mode, etc.).

[0289] It should be noted that the obstacle detection technology disclosed in the present invention can be applied to other fields besides the field of intelligent driving control; for example, it can realize obstacle detection in industrial manufacturing, obstacle detection in indoor areas such as supermarkets, obstacle detection in the security field, etc. The present disclosure does not limit the applicable scenarios of the obstacle detection technology.

[0290] Fig.24 It is a structural schematic diagram of an embodiment of the obstacle detection device disclosed in the present invention. Fig.24 The device in the embodiment includes: an acquisition module 2400, a first determination module 2410, a clustering module 2420 and a second determination module 2430. Optionally, it may also include: a training module 2440.

[0291] The acquisition module 2400 is used to acquire a first disparity map of an environmental image. The environmental image is an image that characterizes the spatial environment information of the smart device during movement. Optionally, the environmental image includes a monocular image. The acquisition module 2400 may include: a first submodule, a second submodule, and a third submodule. The first submodule is used to perform disparity analysis on the monocular image using a convolutional neural network, and obtain a first disparity map of the monocular image based on the output of the convolutional neural network; wherein the convolutional neural network is obtained by training using binocular image samples. The second submodule is used to obtain a first mirror image after mirroring the monocular image, and obtain a disparity map of the first mirror image. The third submodule is used to perform disparity adjustment on the first disparity map of the monocular image according to the disparity map of the first mirror image, and obtain the first disparity map after disparity adjustment. The third submodule may include: a first unit and a second unit. The first unit is used to obtain a second mirror image after mirroring the disparity map of the first mirror image. The second unit is used to perform disparity adjustment on the first disparity image according to the weight distribution map of the first disparity image and the weight distribution map of the second mirror image to obtain the first disparity image after disparity adjustment. The weight distribution map of the first disparity image includes weight values ​​corresponding to each of the multiple disparity values ​​in the first disparity image; the weight distribution map of the second mirror image includes weights corresponding to each of the multiple disparity values ​​in the second mirror image.

[0292] Optionally, the weight distribution map in the present disclosure includes: at least one of a first weight distribution map and a second weight distribution map. The first weight distribution map is a weight distribution map uniformly set for multiple environmental images; the second weight distribution map is a weight distribution map respectively set for different environmental images. The first weight distribution map includes at least two areas divided into left and right columns, and different areas have different weight values.

[0293] Optionally, in the case where the monocular image is the left-eye image to be processed: for any two regions in the first weight distribution map of the first disparity map, the weight value of the region on the right is not less than the weight value of the region on the left; for any two regions in the first weight distribution map of the second mirror image, the weight value of the region on the right is not less than the weight value of the region on the left. For at least one region in the first weight distribution map of the first disparity map, the weight value of the left part of the region is not greater than the weight value of the right part of the region; for at least one region in the first weight distribution map of the second mirror image, the weight value of the left part of the region is not greater than the weight value of the right part of the region.

[0294] Optionally, when the monocular image is the right-eye image to be processed, for any two areas in the first weight distribution map of the first disparity map, the weight value of the area on the left is not less than the weight value of the area on the right; for any two areas in the first weight distribution map of the second mirror image, the weight value of the area on the left is not less than the weight value of the area on the right.

[0295] Optionally, for at least one area in the first weight distribution map of the first disparity image, the weight value of the right part of the area is not greater than the weight value of the left part of the area; for at least one area in the first weight distribution map of the second mirror image, the weight value of the right part of the area is not greater than the weight value of the left part of the area.

[0296] Optionally, the third submodule may further include: a third unit, which is used to set a second weight distribution map of the first disparity map. Specifically, the third unit performs mirror processing on the first disparity map to form a mirror disparity map; and sets the weight value in the second weight distribution map of the first disparity map according to the disparity value in the mirror disparity map of the first disparity map. For example, for a pixel point at any position in the mirror disparity map, when the disparity value of the pixel point at the position meets the first predetermined condition, the third unit sets the weight value of the pixel point at the position in the second weight distribution map of the first disparity map to the first value. When the disparity value of the pixel point does not meet the first predetermined condition, the third unit may set the weight value of the pixel point at the position in the second weight distribution map of the first disparity map to the second value; wherein the first value is greater than the second value. The first predetermined condition may include: the disparity value of the pixel point at the position is greater than the first reference value of the pixel point at the position. The first reference value of the pixel point at the position is set according to the disparity value of the pixel point at the position in the first disparity map and a constant value greater than zero.

[0297] Optionally, the third submodule may further include: a fourth unit. The fourth unit is used to set the fourth unit of the second weight distribution map of the second mirror image. For example, the fourth unit sets the weight value in the second weight distribution map of the second mirror image according to the disparity value in the first disparity map. More specifically, for a pixel point at any position in the second mirror image, if the disparity value of the pixel point at the position in the first disparity map meets the second predetermined condition, the fourth unit sets the weight value of the pixel point at the position in the second weight distribution map of the second mirror image to a third value. If the disparity value of the pixel point at the position in the first disparity map does not meet the second predetermined condition, the fourth unit sets the weight value of the pixel point at the position in the second weight distribution map of the second mirror image to a fourth value; wherein the third value is greater than the fourth value. The second predetermined condition includes: the disparity value of the pixel point at the position in the first disparity map is greater than the second reference value of the pixel point at the position; wherein the second reference value of the pixel point at the position is set according to the disparity value of the pixel point at the position in the mirror disparity map of the first disparity map and a constant value greater than zero.

[0298] Optionally, the second unit can adjust the disparity value in the first disparity image according to the first weight distribution map and the second weight distribution map of the first disparity image; and adjust the disparity value in the second mirror image according to the first weight distribution map and the second weight distribution map of the second mirror image; the second unit merges the first disparity image after disparity adjustment and the second mirror image after disparity value adjustment, and finally obtains the first disparity image after parallax adjustment.

[0299] The specific operations performed by the various parts included in the acquisition module 2400 can refer to the description of S100 in the above method embodiment, and will not be described in detail here.

[0300] The first determination module 2410 is used to determine multiple obstacle pixel areas in the first disparity map of the environment image. The first determination module 2410 may include: a fourth submodule, a fifth submodule and a sixth submodule. The fourth submodule is used to perform edge detection on the first disparity map of the environment image to obtain obstacle edge information. The fifth submodule is used to determine the obstacle area in the first disparity map of the environment image; the sixth submodule is used to determine multiple obstacle pixel columnar areas in the obstacle area of ​​the first disparity map according to the obstacle edge information. The fifth submodule may include: a fifth unit, a sixth unit, a seventh unit and an eighth unit. The fifth unit is used to perform statistical processing on the disparity values ​​of each row of pixels in the first disparity map to obtain statistical information on the disparity values ​​of each row of pixels. The sixth unit is used to determine the statistical disparity map based on the statistical information on the disparity values ​​of each row of pixels; the seventh unit is used to perform a first straight line fitting process on the statistical disparity map, and determine the ground area and the non-ground area according to the result of the first straight line fitting process; the eighth unit is used to determine the obstacle area according to the non-ground area. Among them, the non-ground area includes: a first area above the ground. The non-ground area includes: a first area above the ground and a second area below the ground. The eighth unit can perform a second straight line fitting process on the statistical disparity map, and determine a first target area in the first area whose height above the ground is less than a first predetermined height value according to the result of the second straight line fitting process, and the first target area is an obstacle area; in the case where there is a second area below the ground in the non-ground area, the eighth unit determines a second target area in the second area whose height below the ground is greater than a second predetermined height value, and the second target area is an obstacle area.

[0301] Optionally, the sixth submodule can set the disparity values ​​of the pixel points in the non-obstacle area in the first disparity map and the disparity values ​​of the pixel points at the obstacle edge information to predetermined values; the sixth submodule takes the N pixel points in the column direction of the first disparity map as the traversal unit, traverses the disparity values ​​of the N pixel points on each row starting from the set row of the first disparity map, and determines the target row where the disparity value of the pixel point jumps between the predetermined value and the non-predetermined value; the sixth submodule is used to determine the obstacle pixel columnar area in the obstacle area using the N pixel points in the column direction as the column width and the determined target row as the boundary of the obstacle pixel columnar area in the row direction.

[0302] The specific operations performed by the various parts included in the first determination module 2410 can refer to the description of S110 in the above method embodiment, and will not be described in detail here.

[0303] The clustering module 2420 is used to perform clustering processing on multiple obstacle pixel regions to obtain at least one cluster. For example, the clustering module 2420 can perform clustering processing on multiple obstacle pixel columnar regions. The clustering module 2420 may include a seventh submodule and an eighth submodule. The seventh submodule is used to determine the spatial position information of multiple obstacle pixel columnar regions. The eighth submodule is used to perform clustering processing on multiple obstacle pixel columnar regions according to the spatial position information of multiple obstacle pixel columnar regions. For example, for any obstacle pixel columnar region, the eighth submodule determines the attribute information of the obstacle pixel columnar region according to the pixels contained in the obstacle pixel columnar region, and determines the spatial position information of the obstacle pixel columnar region according to the attribute information of the obstacle pixel columnar region. The attribute information of the obstacle pixel columnar region may include at least one of: pixel columnar region bottom information, pixel columnar region top information, pixel columnar region disparity value, and pixel columnar region column information. The spatial position information of the obstacle pixel columnar region may include: the coordinates of the obstacle pixel columnar region on the horizontal coordinate axis, and the coordinates of the obstacle pixel columnar region on the depth coordinate axis. The spatial position information of the obstacle pixel columnar region may also include: the coordinates of the highest point of the obstacle pixel columnar region on the vertical coordinate axis, and the coordinates of the lowest point of the obstacle pixel columnar region on the vertical coordinate axis; the coordinates of the highest point and the coordinates of the lowest point are used to determine the height of the obstacle. The specific operations performed by the various parts included in the clustering module 2420 can be found in the description of S120 in the above method embodiment, and will not be described in detail here.

[0304] The second determination module 2430 is used to determine the obstacle detection result according to the obstacle pixel area belonging to the same cluster. The second determination module may include: at least one of the ninth submodule and the tenth submodule. The ninth submodule is used to determine the obstacle detection frame in the environment image according to the spatial position information of the obstacle pixel columnar area belonging to the same cluster. The tenth submodule is used to determine the spatial position information of the obstacle according to the spatial position information of the obstacle pixel columnar area belonging to the same cluster. For example, the tenth submodule can determine the distance between multiple obstacle pixel columnar areas and the camera device that generates the environment image according to the spatial position information of multiple obstacle pixel columnar areas belonging to the same cluster; and determine the spatial position information of the obstacle according to the spatial position information of the obstacle pixel columnar area closest to the camera device. The specific operations performed by each part included in the second determination module 2430 can refer to the description of S130 in the above method embodiment, and will not be described in detail here.

[0305] The training module 2440 is a training module for training a convolutional neural network. For example, the training module 2440 inputs one of the binocular image samples into the convolutional neural network to be trained, performs disparity analysis processing via the convolutional neural network, and obtains a disparity map of the left-eye image sample and a disparity map of the right-eye image sample based on the output of the convolutional neural network; the training module 2440 reconstructs the right-eye image according to the disparity map of the left-eye image sample and the right-eye image sample; the training module 2440 reconstructs the left-eye image according to the disparity map of the right-eye image sample and the left-eye image sample; the training module 2440 adjusts the network parameters of the convolutional neural network according to the difference between the reconstructed left-eye image and the left-eye image sample, and the difference between the reconstructed right-eye image and the right-eye image sample. The specific operations performed by the training module 2440 can be found in the above-mentioned Fig. 22 The description will not be described in detail here.

[0306] Fig.25 A schematic diagram of the structure of an embodiment of the intelligent driving control device disclosed in the present invention. Fig.25 The device includes: an acquisition module 2500, an obstacle detection device 2510 and a control module 2520.

[0307] The acquisition module 2500 is used to acquire the environment image of the intelligent device during movement through the image acquisition device set on the intelligent driving device. The obstacle detection device 2510 is used to perform obstacle detection on the environment image and determine the obstacle detection result. The control module 2520 is used to generate and output the vehicle control command according to the obstacle detection result.

[0308] Exemplary Devices

[0309] Fig.26 An exemplary device 2600 suitable for implementing the present disclosure is shown. Device 2600 may be a control system / electronic system configured in an automobile, a mobile terminal (e.g., a smart mobile phone, etc.), a personal computer (PC, such as a desktop computer or a notebook computer, etc.), a tablet computer, a server, etc. Fig.26In the embodiment, the device 2600 includes one or more processors, a communication unit, etc. The one or more processors may be: one or more central processing units (CPUs) 2601, and / or one or more image processing units (GPUs) 2613 using neural networks for visual tracking, etc. The processor may perform various appropriate actions and processes according to executable instructions stored in a read-only memory (ROM) 2602 or executable instructions loaded from a storage part 2608 to a random access memory (RAM) 2603. The communication unit 2612 may include but is not limited to a network card, and the network card may include but is not limited to an IB (Infiniband) network card. The processor may communicate with the read-only memory 2602 and / or the random access memory 2603 to execute executable instructions, be connected to the communication unit 2612 through the bus 2604, and communicate with other target devices through the communication unit 2612, thereby completing the corresponding steps in the present disclosure.

[0310] The operations performed by the above instructions can refer to the relevant descriptions in the above method embodiments, and will not be described in detail here. In addition, various programs and data required for device operation can also be stored in RAM2603. CPU2601, ROM2602 and RAM2603 are connected to each other via bus 2604.

[0311] In the case of RAM2603, ROM2602 is an optional module. RAM2603 stores executable instructions, or writes executable instructions into ROM2602 at runtime, and the executable instructions enable the central processing unit 2601 to perform the steps included in the above-mentioned object segmentation method. The input / output (I / O) interface 2605 is also connected to the bus 2604. The communication unit 2612 can be integrated or can be configured to have multiple sub-modules (for example, multiple IB network cards) and connected to the bus respectively.

[0312] The following components are connected to the I / O interface 2605: an input section 2606 including a keyboard, a mouse, etc.; an output section 2607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 2608 including a hard disk, etc.; and a communication section 2609 including a network interface card such as a LAN card, a modem, etc. The communication section 2609 performs communication processing via a network such as the Internet. A drive 2610 is also connected to the I / O interface 2605 as needed. A removable medium 2611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 2610 as needed, so that a computer program read therefrom is installed in the storage section 2608 as needed.

[0313] It should be noted that if Fig.26The architecture shown is only an optional implementation method. In the specific practice, the above can be modified according to actual needs. Fig.26 The number and type of components can be selected, deleted, added or replaced; different functional components can be set up by separate settings or integrated settings. For example, GPU2613 and CPU2601 can be set up separately, and GPU2613 can be integrated into CPU2601. The communication unit can be set up separately or integrated into CPU2601 or GPU2613. These replaceable implementations all fall within the protection scope of the present disclosure.

[0314] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes program codes for executing the steps shown in the flowchart. The program code may include instructions corresponding to the steps in the method provided by the present disclosure.

[0315] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 2609, and / or installed from the removable medium 2611. When the computer program is executed by the central processing unit (CPU) 2601, the instructions recorded in the present disclosure for implementing the above corresponding steps are executed.

[0316] In one or more optional embodiments, the embodiments of the present disclosure further provide a computer program product for storing computer-readable instructions, which, when executed, enable a computer to execute the obstacle detection method or intelligent driving control method described in any of the above embodiments.

[0317] The computer program product may be implemented in hardware, software or a combination thereof. In one optional example, the computer program product is embodied as a computer storage medium, and in another optional example, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0318] In one or more optional embodiments, the embodiments of the present disclosure also provide another visual tracking method and a neural network training method and corresponding devices and electronic devices, computer storage media, computer programs and computer program products, wherein the method includes: a first device sends an obstacle detection indication or an intelligent driving control indication to a second device, and the indication enables the second device to execute the obstacle detection method or the intelligent driving control method in any possible embodiment described above; the first device receives the obstacle detection result or the intelligent driving control result sent by the second device.

[0319] In some embodiments, the visual obstacle detection indication or intelligent driving control indication may be specifically a calling instruction. The first device may instruct the second device to perform an obstacle detection operation or an intelligent driving control operation by calling. Accordingly, in response to receiving the calling instruction, the second device may execute the steps and / or processes in any embodiment of the above-mentioned obstacle detection method or intelligent driving control method.

[0320] It should be understood that the terms "first", "second", etc. in the embodiments of the present disclosure are only for distinction and should not be understood as limitations on the embodiments of the present disclosure. It should also be understood that in the present disclosure, "multiple" may refer to two or more, and "at least one" may refer to one, two or more. It should also be understood that any component, data or structure mentioned in the present disclosure can generally be understood as one or more in the absence of clear limitations or contrary revelations given in the context. It should also be understood that the description of each embodiment in the present disclosure emphasizes the differences between the embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated one by one.

[0321] The method and apparatus, electronic device and computer-readable storage medium of the present disclosure may be implemented in many ways. For example, the method and apparatus, electronic device and computer-readable storage medium of the present disclosure may be implemented by software, hardware, firmware or any combination of software, hardware and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure. The description of the present disclosure is given for the sake of example and description, and is not exhaustive or limits the present disclosure to the disclosed form. Many modifications and changes are obvious to those of ordinary skill in the art. The selection and description of the implementation mode is to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand that the embodiments of the present disclosure can be designed to be suitable for specific purposes. Various implementation modes with various modifications.

Claims

1. An obstacle detection method, characterized in that: include: Acquire a first disparity map of an environment image, where the environment image is an image representing spatial environment information in which the smart device is located during movement; Determining a plurality of obstacle pixel regions in the first disparity map of the environment image, wherein the obstacle pixel regions include obstacle pixel columnar regions; Performing clustering processing on the plurality of obstacle pixel regions to obtain at least one cluster, wherein each cluster in the at least one cluster corresponds to an obstacle; Determine the obstacle detection result according to the obstacle pixel areas belonging to the same cluster; The clustering of the plurality of obstacle pixel regions comprises: Determining spatial position information of the plurality of obstacle pixel columnar regions; performing clustering processing on the plurality of obstacle pixel columnar regions according to spatial position information of the plurality of obstacle pixel columnar regions; The step of determining a plurality of obstacle pixel areas in the first disparity map of the environment image comprises: Performing edge detection on the first disparity map of the environment image to obtain obstacle edge information; Determining an obstacle area in a first disparity map of the environment image; According to the obstacle edge information, a plurality of obstacle pixel columnar regions are determined in the obstacle region of the first disparity map.

2. The method according to claim 1, characterized in that The environment image includes a monocular image; After obtaining the first disparity map of the environment image, the method further includes: After mirroring the monocular image, a first mirror image is obtained, and a disparity map of the first mirror image is acquired; According to the disparity map of the first mirror image, performing disparity adjustment on the first disparity map of the monocular image to obtain a first disparity map after disparity adjustment; The step of determining a plurality of obstacle pixel areas in the first disparity map of the environment image comprises: A plurality of obstacle pixel regions are determined in the first disparity map after disparity adjustment.

3. The method according to claim 2, characterized in that The step of performing disparity adjustment on the first disparity map of the monocular image according to the disparity map of the first mirror image to obtain the first disparity map after disparity adjustment includes: After mirroring the disparity map of the first mirror image, a second mirror image is obtained; According to the weight distribution map of the first disparity map and the weight distribution map of the second mirror image, performing disparity adjustment on the first disparity map to obtain a first disparity map after disparity adjustment; The weight distribution map of the first disparity map includes weight values ​​corresponding to each of the multiple disparity values ​​in the first disparity map; and the weight distribution map of the second mirror image includes weights corresponding to each of the multiple disparity values ​​in the second mirror image.

4. The method according to claim 3, characterized in that The weight distribution map includes: a first weight distribution map, and / or a second weight distribution map; The first weight distribution map is a weight distribution map uniformly set for multiple environment images; The second weight distribution map is a weight distribution map set for different environment images respectively.

5. The method according to claim 4, characterized in that The first weight distribution diagram includes at least two areas in left and right columns, and different areas have different weight values.

6. The method according to claim 5, characterized in that When the monocular image is a left-eye image to be processed: For any two regions in the first weight distribution map of the first disparity map, the weight value of the region located on the right is not less than the weight value of the region located on the left; For any two regions in the first weight distribution diagram of the second mirror image, the weight value of the region located on the right is not less than the weight value of the region located on the left.

7. The method according to claim 6, characterized in that: For at least one region in the first weight distribution map of the first disparity map, a weight value of a left portion of the region is not greater than a weight value of a right portion of the region; For at least one region in the first weight distribution map of the second mirror image, a weight value of a left portion of the region is not greater than a weight value of a right portion of the region.

8. The method according to claim 5, characterized in that When the monocular image is a right-eye image to be processed: For any two regions in the first weight distribution map of the first disparity map, the weight value of the region on the left is not less than the weight value of the region on the right; For any two regions in the first weight distribution diagram of the second mirror image, the weight value of the region located on the left is not less than the weight value of the region located on the right.

9. The method according to claim 8, characterized in that: For at least one region in the first weight distribution map of the first disparity map, a weight value of a right portion of the region is not greater than a weight value of a left portion of the region; For at least one region in the first weight distribution map of the second mirror image, a weight value of a right portion of the region is not greater than a weight value of a left portion of the region.

10. The method according to any one of claims 4 to 9, characterized in that The second weight distribution map of the first disparity map is set in a manner including: Performing mirror processing on the first disparity map to form a mirror disparity map; The weight values ​​in the second weight distribution map of the first disparity map are set according to the disparity values ​​in the mirror disparity map of the first disparity map.

11. The method according to claim 10, characterized in that The step of setting the weight value in the second weight distribution map of the first disparity map according to the disparity value in the mirror disparity map of the first disparity map comprises: For a pixel point at any position in the mirror disparity map, when the disparity value of the pixel point at the position satisfies a first predetermined condition, the weight value of the pixel point at the position in the second weight distribution map of the first disparity map is set to a first value.

12. The method according to claim 11, characterized in that When the disparity value of the pixel point does not satisfy the first predetermined condition, the method further includes: The weight value of the pixel at the position in the second weight distribution map of the first disparity map is set to a second value; wherein the first value is greater than the second value.

13. The method according to claim 11, characterized in that The first predetermined condition includes: the disparity value of the pixel point at the position is greater than the first reference value of the pixel point at the position; The first reference value of the pixel at the position is set according to the disparity value of the pixel at the position in the first disparity map and a constant value greater than zero.

14. The method according to any one of claims 4 to 9, characterized in that The second weight distribution diagram of the second mirror image is set in the following manner: The weight values ​​in the second weight distribution map of the second mirror image are set according to the disparity values ​​in the first disparity map.

15. The method according to claim 14, characterized in that The step of setting the weight value in the second weight distribution map of the second mirror image according to the disparity value in the first disparity map comprises: For a pixel point at any position in the second mirror image, if the disparity value of the pixel point at the position in the first disparity map satisfies a second predetermined condition, the weight value of the pixel point at the position in the second weight distribution map of the second mirror image is set to a third value.

16. The method according to claim 15, characterized in that When the disparity value of the pixel point at the position in the first disparity map does not satisfy a second predetermined condition, the method further includes: The weight value of the pixel point at the position in the second weight distribution map of the second mirror image is set to a fourth value; wherein the third value is greater than the fourth value.

17. The method according to claim 15, characterized in that The second predetermined condition includes: the disparity value of the pixel at the position in the first disparity map is greater than a second reference value of the pixel at the position; The second reference value of the pixel at the position is set according to the disparity value of the pixel at the position in the mirror disparity map of the first disparity map and a constant value greater than zero.

18. The method according to any one of claims 4 to 9, characterized in that: The step of performing parallax adjustment on the first parallax image according to the weight distribution map of the first parallax image and the weight distribution map of the second mirror image to obtain the first parallax image after the parallax adjustment comprises: adjusting a disparity value in the first disparity map according to a first weight distribution map and a second weight distribution map of the first disparity map; adjusting the parallax value in the second mirror image according to the first weight distribution map and the second weight distribution map of the second mirror image; After merging the first disparity image after disparity adjustment and the second mirror image after disparity value adjustment, the first disparity image after disparity adjustment is finally obtained.

19. The method according to claim 1, characterized in that The environment image includes a monocular image; The step of acquiring a first disparity map of an environment image comprises: Performing disparity analysis on the monocular image using a convolutional neural network, and obtaining a first disparity map of the monocular image based on an output of the convolutional neural network; Wherein, the convolutional neural network is obtained by training using binocular image samples.

20. The method according to claim 19, characterized in that The training process of the convolutional neural network includes: Inputting one of the binocular image samples into a convolutional neural network to be trained, performing disparity analysis processing via the convolutional neural network, and obtaining a disparity map of the left-eye image sample and a disparity map of the right-eye image sample based on an output of the convolutional neural network; Reconstructing the right-eye image according to the disparity map of the left-eye image samples and the right-eye image samples; Reconstructing the left-eye image according to the disparity map of the right-eye image samples and the left-eye image samples; According to the difference between the reconstructed left-eye image and the left-eye image sample, and the difference between the reconstructed right-eye image and the right-eye image sample, the network parameters of the convolutional neural network are adjusted.

21. The method according to claim 1, characterized in that The determining of the obstacle area in the first disparity map of the environment image includes: Performing statistical processing on the disparity values ​​of each row of pixels in the first disparity map to obtain statistical information on the disparity values ​​of each row of pixels; Determine a statistical disparity map based on statistical information of disparity values ​​of each row of pixels; Performing a first straight line fitting process on the statistical disparity map, and determining a ground area and a non-ground area according to a result of the first straight line fitting process; An obstacle area is determined according to the non-ground area.

22. The method according to claim 21, characterized in that The non-ground area includes: a first area higher than the ground; or, the non-ground area includes: a first area higher than the ground and a second area lower than the ground.

23. The method according to claim 22, characterized in that The step of determining an obstacle area according to the non-ground area includes: performing a second straight-line fitting process on the statistical disparity map, and determining, according to a result of the second straight-line fitting process, a first target area in the first area whose height above the ground is less than a first predetermined height value, wherein the first target area is an obstacle area; In the case that there is a second area below the ground in the non-ground area, a second target area whose height below the ground is greater than a second predetermined height value is determined in the second area, and the second target area is an obstacle area.

24. The method according to claim 1, characterized in that The determining, in the obstacle region of the first disparity map according to the obstacle edge information, a plurality of obstacle pixel columnar regions comprises: Setting the disparity values ​​of the pixels in the non-obstacle area in the first disparity map and the disparity values ​​of the pixels at the obstacle edge information to predetermined values; Taking N pixels in the column direction of the first disparity map as traversal units, traversing the disparity values ​​of N pixels on each row starting from a set row of the first disparity map, and determining a target row where the disparity value of the pixel point jumps between the predetermined value and the non-predetermined value; N is a positive integer; The obstacle pixel column region in the obstacle region is determined by taking N pixel points in the column direction as the column width and taking the determined target row as the boundary of the obstacle pixel column region in the row direction.

25. The method according to claim 1, characterized in that The determining of the spatial position information of the plurality of obstacle pixel columnar regions includes: For any obstacle pixel column region, attribute information of the obstacle pixel column region is determined according to pixels included in the obstacle pixel column region, and spatial position information of the obstacle pixel column region is determined according to the attribute information of the obstacle pixel column region.

26. The method according to claim 25, characterized in that The attribute information of the obstacle pixel column region includes at least one of: pixel column region bottom information, pixel column region top information, pixel column region disparity value, and pixel column region column information.

27. The method according to claim 25, characterized in that The spatial position information of the obstacle pixel column region includes: the coordinates of the obstacle pixel column region on the horizontal coordinate axis and the coordinates of the obstacle pixel column region on the depth coordinate axis.

28. The method according to claim 27, characterized in that The spatial position information of the obstacle pixel column area also includes: the coordinates of the highest point of the obstacle pixel column area on the vertical coordinate axis, and the coordinates of the lowest point of the obstacle pixel column area on the vertical coordinate axis; The highest point coordinates and the lowest point coordinates are used to determine the height of the obstacle.

29. The method according to any one of claims 1 to 9, characterized in that The obstacle pixel area includes: Obstacle pixel column area; The step of determining the obstacle detection result according to the obstacle pixel regions belonging to the same cluster includes: Determining an obstacle detection frame in the environment image according to spatial position information of a columnar region of obstacle pixels belonging to the same cluster; and / or The spatial position information of the obstacle is determined according to the spatial position information of the obstacle pixel columnar area belonging to the same cluster.

30. The method according to claim 29, characterized in that The step of determining the spatial position information of the obstacle according to the spatial position information of the obstacle pixel columnar regions belonging to the same cluster includes: Determining, according to spatial position information of a plurality of obstacle pixel columnar regions belonging to the same cluster, distances between the plurality of obstacle pixel columnar regions and a camera device that generates the environment image; The spatial position information of the obstacle is determined according to the spatial position information of the obstacle pixel column area closest to the camera device.

31. An intelligent driving control method, characterized in that: include: Acquiring an image of the environment of the smart driving device during movement through an image acquisition device provided on the smart driving device; Using the method according to any one of claims 1 to 30, performing obstacle detection on the acquired environment image to determine the obstacle detection result; A control instruction is generated and outputted according to the obstacle detection result.

32. An obstacle detection device, characterized in that: include: An acquisition module, configured to acquire a first disparity map of an environment image, wherein the environment image is an image representing spatial environment information in which the smart device is located during movement; A first determining module, configured to determine a plurality of obstacle pixel regions in a first disparity map of the environment image, wherein the obstacle pixel regions include obstacle pixel columnar regions; A clustering module, configured to perform clustering processing on the plurality of obstacle pixel regions to obtain at least one cluster, wherein each cluster in the at least one cluster corresponds to one obstacle; A second determination module is used to determine the obstacle detection result according to the obstacle pixel areas belonging to the same cluster; Wherein, the clustering module includes: A seventh submodule is used to determine the spatial position information of the plurality of obstacle pixel columnar regions; An eighth submodule, configured to perform clustering processing on the plurality of obstacle pixel columnar regions according to spatial position information of the plurality of obstacle pixel columnar regions; The first determining module includes: A fourth submodule is used to perform edge detection on the first disparity map of the environment image to obtain obstacle edge information; A fifth submodule, configured to determine an obstacle area in the first disparity map of the environment image; The sixth submodule is configured to determine a plurality of obstacle pixel columnar regions in the obstacle region of the first disparity map according to the obstacle edge information.

33. The device according to claim 32, characterized in that The acquisition module also includes: The second submodule is used to obtain a first mirror image after performing mirror processing on the monocular image in the environment image, and obtain a disparity map of the first mirror image; A third submodule is used to perform disparity adjustment on the first disparity map of the monocular image according to the disparity map of the first mirror image to obtain the first disparity map after disparity adjustment; The first determining module is further configured to: A plurality of obstacle pixel regions are determined in the first disparity map after disparity adjustment.

34. The device according to claim 33, characterized in that The third submodule comprises: A first unit is used for performing mirror processing on the disparity map of the first mirror image to obtain a second mirror image; A second unit is configured to perform disparity adjustment on the first disparity map according to the weight distribution map of the first disparity map and the weight distribution map of the second mirror image to obtain a first disparity map after disparity adjustment; The weight distribution map of the first disparity map includes weight values ​​corresponding to each of the multiple disparity values ​​in the first disparity map; and the weight distribution map of the second mirror image includes weights corresponding to each of the multiple disparity values ​​in the second mirror image.

35. The device according to claim 34, characterized in that The weight distribution map includes: a first weight distribution map, and / or a second weight distribution map; The first weight distribution map is a weight distribution map uniformly set for multiple environment images; The second weight distribution map is a weight distribution map set for different environment images respectively.

36. The device according to claim 35, characterized in that The first weight distribution diagram includes at least two areas in left and right columns, and different areas have different weight values.

37. The device according to claim 36, characterized in that When the monocular image is a left-eye image to be processed: For any two regions in the first weight distribution map of the first disparity map, the weight value of the region located on the right is not less than the weight value of the region located on the left; For any two regions in the first weight distribution diagram of the second mirror image, the weight value of the region located on the right is not less than the weight value of the region located on the left.

38. The device according to claim 37, characterized in that: For at least one region in the first weight distribution map of the first disparity map, a weight value of a left portion of the region is not greater than a weight value of a right portion of the region; For at least one region in the first weight distribution map of the second mirror image, a weight value of a left portion of the region is not greater than a weight value of a right portion of the region.

39. The device according to claim 36, characterized in that When the monocular image is a right-eye image to be processed: For any two regions in the first weight distribution map of the first disparity map, the weight value of the region on the left is not less than the weight value of the region on the right; For any two regions in the first weight distribution diagram of the second mirror image, the weight value of the region located on the left is not less than the weight value of the region located on the right.

40. The device according to claim 39, characterized in that: For at least one region in the first weight distribution map of the first disparity map, a weight value of a right portion of the region is not greater than a weight value of a left portion of the region; For at least one region in the first weight distribution map of the second mirror image, a weight value of a right portion of the region is not greater than a weight value of a left portion of the region.

41. The device according to any one of claims 35 to 40, characterized in that The third submodule further includes: a third unit for setting a second weight distribution map of the first disparity map; The third unit performs mirror processing on the first disparity map to form a mirror disparity map; and sets a weight value in a second weight distribution map of the first disparity map according to a disparity value in the mirror disparity map of the first disparity map.

42. The device according to claim 41, characterized in that The third unit is further configured to, for a pixel point at any position in the mirror disparity map, set a weight value of the pixel point at the position in the second weight distribution map of the first disparity map to a first value when the disparity value of the pixel point at the position satisfies a first predetermined condition.

43. The device according to claim 42, characterized in that The third unit is further configured to, when the disparity value of the pixel point does not satisfy the first predetermined condition, set the weight value of the pixel point at the position in the second weight distribution map of the first disparity map to a second value; wherein the first value is greater than the second value.

44. The device according to claim 42, characterized in that The first predetermined condition includes: the disparity value of the pixel point at the position is greater than the first reference value of the pixel point at the position; The first reference value of the pixel at the position is set according to the disparity value of the pixel at the position in the first disparity map and a constant value greater than zero.

45. The device according to any one of claims 35 to 40, characterized in that The third submodule further includes: a fourth unit for setting a second weight distribution map of the second mirror image; The fourth unit sets the weight value in the second weight distribution map of the second mirror image according to the disparity value in the first disparity map.

46. ​​The device according to claim 45, characterized in that The fourth unit is further configured to: For a pixel point at any position in the second mirror image, if the disparity value of the pixel point at the position in the first disparity map satisfies a second predetermined condition, the weight value of the pixel point at the position in the second weight distribution map of the second mirror image is set to a third value.

47. The device according to claim 46, characterized in that The fourth unit is further configured to, when the disparity value of the pixel point at the position in the first disparity map does not satisfy a second predetermined condition, set the weight value of the pixel point at the position in the second weight distribution map of the second mirror image to a fourth value; wherein the third value is greater than the fourth value.

48. The device according to claim 46, characterized in that The second predetermined condition includes: the disparity value of the pixel at the position in the first disparity map is greater than a second reference value of the pixel at the position; The second reference value of the pixel at the position is set according to the disparity value of the pixel at the position in the mirror disparity map of the first disparity map and a constant value greater than zero.

49. The device according to any one of claims 35 to 40, characterized in that The second unit is further used for: adjusting a disparity value in the first disparity map according to a first weight distribution map and a second weight distribution map of the first disparity map; adjusting the parallax value in the second mirror image according to the first weight distribution map and the second weight distribution map of the second mirror image; After merging the first disparity image after disparity adjustment and the second mirror image after disparity value adjustment, the first disparity image after disparity adjustment is finally obtained.

50. The device according to claim 32, characterized in that The environment image includes a monocular image; The acquisition module comprises: A first submodule is used to perform disparity analysis processing on the monocular image using a convolutional neural network, and obtain a first disparity map of the monocular image based on an output of the convolutional neural network; Wherein, the convolutional neural network is obtained by training using binocular image samples.

51. The device according to claim 50, characterized in that The apparatus further comprises: a training module for training a convolutional neural network, wherein the training module is further used to: Inputting one of the binocular image samples into a convolutional neural network to be trained, performing disparity analysis processing via the convolutional neural network, and obtaining a disparity map of the left-eye image sample and a disparity map of the right-eye image sample based on an output of the convolutional neural network; Reconstructing the right-eye image according to the disparity map of the left-eye image samples and the right-eye image samples; Reconstructing the left-eye image according to the disparity map of the right-eye image samples and the left-eye image samples; According to the difference between the reconstructed left-eye image and the left-eye image sample, and the difference between the reconstructed right-eye image and the right-eye image sample, the network parameters of the convolutional neural network are adjusted.

52. The device according to claim 32, characterized in that The fifth submodule comprises: A fifth unit is used to perform statistical processing on the disparity values ​​of each row of pixel points in the first disparity map to obtain statistical information on the disparity values ​​of each row of pixel points; A sixth unit, configured to determine a statistical disparity map based on statistical information of disparity values ​​of each row of pixels; A seventh unit is configured to perform a first straight line fitting process on the statistical disparity map, and determine a ground area and a non-ground area according to a result of the first straight line fitting process; The eighth unit is used to determine an obstacle area according to the non-ground area.

53. The device according to claim 52, characterized in that The non-ground area includes: a first area higher than the ground; or, the non-ground area includes: a first area higher than the ground and a second area lower than the ground.

54. The device according to claim 53, characterized in that The eighth unit is further used for: performing a second straight-line fitting process on the statistical disparity map, and determining, according to a result of the second straight-line fitting process, a first target area in the first area whose height above the ground is less than a first predetermined height value, wherein the first target area is an obstacle area; In the case that there is a second area below the ground in the non-ground area, a second target area whose height below the ground is greater than a second predetermined height value is determined in the second area, and the second target area is an obstacle area.

55. The device according to claim 52, characterized in that The sixth submodule is further used for: Setting the disparity values ​​of the pixels in the non-obstacle area in the first disparity map and the disparity values ​​of the pixels at the obstacle edge information to predetermined values; Taking N pixels in the column direction of the first disparity map as traversal units, traversing the disparity values ​​of N pixels on each row starting from a set row of the first disparity map, and determining a target row where the disparity value of the pixel point jumps between the predetermined value and the non-predetermined value; N is a positive integer; The obstacle pixel column region in the obstacle region is determined by taking N pixel points in the column direction as the column width and taking the determined target row as the boundary of the obstacle pixel column region in the row direction.

56. The device according to claim 32, characterized in that The eighth submodule is further used for: For any obstacle pixel column region, attribute information of the obstacle pixel column region is determined according to pixels included in the obstacle pixel column region, and spatial position information of the obstacle pixel column region is determined according to the attribute information of the obstacle pixel column region.

57. The device according to claim 56, characterized in that The attribute information of the obstacle pixel column region includes at least one of: pixel column region bottom information, pixel column region top information, pixel column region disparity value, and pixel column region column information.

58. The device according to claim 32, characterized in that The spatial position information of the obstacle pixel column region includes: the coordinates of the obstacle pixel column region on the horizontal coordinate axis and the coordinates of the obstacle pixel column region on the depth coordinate axis.

59. The device according to claim 58, characterized in that The spatial position information of the obstacle pixel column area also includes: the coordinates of the highest point of the obstacle pixel column area on the vertical coordinate axis, and the coordinates of the lowest point of the obstacle pixel column area on the vertical coordinate axis; The highest point coordinates and the lowest point coordinates are used to determine the height of the obstacle.

60. The device according to any one of claims 32 to 40, characterized in that The obstacle pixel area includes: Obstacle pixel column area; The second determining module comprises: A ninth submodule, configured to determine an obstacle detection frame in the environment image according to spatial position information of a columnar region of obstacle pixels belonging to the same cluster; and / or The tenth submodule is used to determine the spatial position information of the obstacle according to the spatial position information of the obstacle pixel columnar area belonging to the same cluster.

61. The device according to claim 60, characterized in that The tenth submodule is further configured to: Determining, according to spatial position information of a plurality of obstacle pixel columnar regions belonging to the same cluster, distances between the plurality of obstacle pixel columnar regions and a camera device that generates the environment image; The spatial position information of the obstacle is determined according to the spatial position information of the obstacle pixel column area closest to the camera device.

62. An intelligent driving control device, characterized in that: include: An acquisition module, used to acquire an image of the environment of the smart device during movement through an image acquisition device provided on the smart driving device; Using the device described in any one of claims 32 to 61, performing obstacle detection on the environment image to determine an obstacle detection result; A control module is used to generate and output control instructions according to the obstacle detection result.

63. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute a computer program stored in the memory, and wherein when the computer program is executed, the method according to any one of claims 1 to 31 is implemented.

64. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method described in any one of claims 1 to 31 is implemented.

65. A computer program product, characterized in that The method comprises computer instructions, which, when executed in a processor of a device, implement the method of any one of claims 1 to 31.

Images