Mesh obstacle identification method and device, electronic equipment and computer storage medium

Abstract

This invention relates to a method, apparatus, electronic device, and computer storage medium for identifying mesh obstacles. The method includes acquiring an image to be identified, wherein the image includes mesh obstacles; inputting the image to be identified into a well-trained semantic segmentation prediction model, outputting a semantic segmentation map of the image to be identified; acquiring a depth map of the image to be identified; fusing the semantic segmentation map and the depth map, and performing pixel analysis on the fused image to determine the depth information of the mesh obstacles. This invention improves the accuracy of mesh obstacle recognition by autonomous driving equipment by fusing the semantic segmentation map and the depth map, utilizing the precise classification characteristics of the semantic segmentation map to obtain accurate depth information of the mesh obstacles, thus ensuring driving safety.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, electronic device, and computer storage medium for identifying mesh obstacles. Background Technology

[0002] Autonomous driving technology can be divided into three main modules: perception, cognition, and control. First, it needs to accurately perceive the environment, then process the information, and finally send instructions to the vehicle's control system to achieve specific functions.

[0003] In the perception section, there are a large number of sensors that work together to acquire as much useful information as possible so that the vehicle can move along the correct path. These include LiDAR, millimeter-wave radar, ultrasonic radar, cameras, inertial navigation (IMU), wheeled odometers, etc. Among them, the most operable and scalable sensor is the camera, because it is closest to the principle of human eyes recognizing the environment. Therefore, cameras are widely used in autonomous driving and have attracted a large number of scholars and engineers to study them.

[0004] Typically, autonomous vehicles are required to operate during the day or in well-lit conditions. Furthermore, with advancements in hardware technology, computing power has increased significantly, and onboard cameras are capable of handling environmental perception tasks under most conditions. However, for mesh obstacle recognition, the mesh targets are extremely small, and the mesh is often horizontally distributed, resulting in no significant parallax between the left and right cameras of a binocular camera. This makes it difficult to identify mesh obstacles, posing a safety hazard. Summary of the Invention

[0005] In view of this, it is necessary to provide a method, device, electronic device and computer storage medium for identifying mesh obstacles, so as to solve the problem of low accuracy in identifying mesh obstacles in the prior art, which leads to collisions of unmanned driving equipment.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a method for identifying mesh obstacles, comprising:

[0007] Acquire an image to be identified, wherein the image to be identified includes a mesh of obstacles;

[0008] The image to be identified is input into a fully trained semantic segmentation prediction model, which outputs a semantic segmentation map of the image to be identified.

[0009] Obtain the depth map of the image to be identified;

[0010] The semantic segmentation map and the depth map are fused together, and pixel analysis is performed on the fused image to determine the depth information of the mesh obstacles.

[0011] Furthermore, the fully trained semantic segmentation prediction model is trained based on the PSPNet neural network;

[0012] The structure of the PSPNet neural network includes a feature extraction subnetwork, a pooling subnetwork, and a convolutional subnetwork.

[0013] Furthermore, the training process of the semantic segmentation prediction model includes:

[0014] Obtain an image set including mesh obstacles, and label the image set with classification tags to obtain a classification result set;

[0015] The image set and the classification results corresponding to each image are combined to form a dataset, wherein the dataset includes a training set, a test set and a prediction set;

[0016] The training set is input into the PSPNet neural network for training. After the preset loss condition is reached, the trained model parameters are obtained, and the trained model parameters are loaded based on the PSPNet neural network to complete the training of the semantic segmentation prediction model.

[0017] Specifically, inputting the training set into the PSPNet neural network for training includes:

[0018] The feature extraction subnetwork is used to extract the first image feature layer from the training set;

[0019] The pooling subnetwork is used to perform pooling operations at different scales on the first image feature layer to obtain the second image feature layer.

[0020] The number of feature layers and channels of the second image feature layer are adjusted using the convolutional subnetwork to make the output image the same size as the input image.

[0021] Furthermore, obtaining the depth map of the image to be identified includes:

[0022] Obtain the calibration parameters of the depth camera, and correct the image to be identified based on the calibration parameters;

[0023] The corrected image is matched, and the depth of each pixel in the image to be identified is calculated based on the matching results to obtain the depth map of the image to be identified.

[0024] Furthermore, the step of image fusion of the semantic segmentation map and the depth map includes:

[0025] Construct a camera coordinate system based on the depth camera;

[0026] The semantic segmentation map and the depth map are fused in the camera coordinate system to obtain a fused image.

[0027] Furthermore, the pixel analysis of the fused image to determine the depth information of the mesh obstacles includes:

[0028] Obtain the histogram corresponding to the fused image, perform pixel statistics on the histogram, and determine the depth information of the mesh obstacle based on the statistical results.

[0029] Furthermore, the method also includes:

[0030] The depth map is filled and repaired based on the depth information.

[0031] In a second aspect, the present invention also provides a mesh obstacle recognition device, comprising:

[0032] The first acquisition module is used to acquire an image to be identified, wherein the image to be identified includes a mesh obstacle;

[0033] The output module is used to input the image to be recognized into a fully trained semantic segmentation prediction model and output a semantic segmentation map of the image to be recognized.

[0034] The second acquisition module is used to acquire the depth map of the image to be identified;

[0035] The determination module is used to perform image fusion of the semantic segmentation map and the depth map, and to perform pixel analysis on the fused image to determine the depth information of the mesh obstacle.

[0036] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the above-described mesh obstacle recognition method.

[0037] Fourthly, the present invention also provides a computer storage medium, which stores a computer program that, when executed by a processor, implements the steps in a mesh obstacle recognition method.

[0038] The beneficial effects of using the above embodiments are:

[0039] This invention acquires images to be identified in real time using a camera, then uses semantic segmentation technology to obtain semantic segmentation maps of the images to be identified, enabling accurate detection of mesh obstacles in real time. The depth map generated by binocular stereo imaging is then corrected and supplemented based on the set of semantic segmentation maps, improving the recognition accuracy of autonomous driving equipment for mesh obstacles. This allows robots or autonomous vehicles to have a comprehensive perception of the environment in complex environments, enabling precise obstacle avoidance and further improving driving safety. Attached Figure Description

[0040] Figure 1 A flowchart illustrating an embodiment of the mesh obstacle recognition method provided by the present invention;

[0041] Figure 2 A reference image of an image to be identified is provided according to an embodiment of the present invention;

[0042] Figure 3 A reference diagram of a semantic segmentation map of an image to be identified, provided in an embodiment of the present invention;

[0043] Figure 4 This is an overall framework diagram of PSPNet provided in an embodiment of the present invention;

[0044] Figure 5 This is a label manufacturing effect diagram provided in one embodiment of the present invention;

[0045] Figure 6 A depth map of an image to be identified is provided in one embodiment of the present invention;

[0046] Figure 7 This is a fused image obtained by fusing a semantic segmentation map and a depth map of an image to be identified, as provided in an embodiment of the present invention.

[0047] Figure 8 This invention provides a histogram corresponding to a portion of a fused image region as an embodiment of the present invention.

[0048] Figure 9 A schematic diagram of a structure of an embodiment of the mesh obstacle recognition device provided by the present invention;

[0049] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0051] In the description of this invention, "a plurality of" means two or more, unless otherwise expressly specified. The reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0052] Autonomous vehicles are generally required to operate during the day or in well-lit conditions. Furthermore, with advancements in hardware technology and increasingly powerful computing capabilities, depth cameras in autonomous driving systems can meet environmental perception requirements under most conditions. However, for mesh obstacles, the mesh objects are often very small and horizontally distributed, resulting in a lack of significant parallax for depth cameras, making it difficult to identify mesh obstacles. Semantic segmentation maps, on the other hand, can represent the regions where objects are located within an image. Therefore, by combining depth maps and semantic segmentation maps, mesh obstacles can be effectively identified.

[0053] This invention provides a method, apparatus, electronic device, and computer storage medium for identifying mesh obstacles, which will be described below.

[0054] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the mesh obstacle recognition method provided by the present invention. A specific embodiment of the present invention discloses a mesh obstacle recognition method, comprising:

[0055] Step S101: Obtain the image to be identified, wherein the image to be identified includes mesh obstacles;

[0056] Step S102: Input the image to be recognized into the fully trained semantic segmentation prediction model, and output the semantic segmentation map of the image to be recognized;

[0057] Step S103: Obtain the depth map of the image to be identified;

[0058] Step S104: Perform image fusion of the semantic segmentation map and the depth map, and perform pixel analysis on the fused image to determine the depth information of the mesh obstacles.

[0059] The images to be identified include those containing mesh obstacles. It's understood that autonomous driving devices can fulfill environmental perception tasks under most conditions during autonomous driving. Autonomous driving devices include, but are not limited to, robots and driverless vehicles. However, mesh obstacles are difficult to identify because they are extremely small and often horizontally distributed. Therefore, it is necessary to process the mesh obstacles in the field of vision of the autonomous driving device to facilitate its autonomous driving.

[0060] Specifically, the image to be identified can be acquired using the depth camera of an autonomous vehicle, and then input into a well-trained semantic segmentation prediction model, which outputs a semantic segmentation map for the image to be identified. Please refer to the following for details. Figure 2 , Figure 3 , Figure 2 This is a reference image of an image to be identified according to an embodiment of the present invention. Figure 3 This is a reference diagram of a semantic segmentation map of an image to be identified, provided as an embodiment of the present invention. It is understood that semantic representation progresses from concrete to abstract; semantic segmentation refers to instructing a computer to segment an image based on its semantics. In the image domain, semantics refers to the content of the image. Therefore, the semantic segmentation map of the image to be identified is a map presented after classifying the content in the image.

[0061] Understandably, because the mesh seen by a depth camera lacks significant parallax, the depth information of the mesh obstacles acquired by the depth camera is inaccurate, while the semantic segmentation map clearly presents the content of each classified region. Therefore, image fusion can be performed between the depth map and the semantic map of the image to be identified captured by the depth camera, followed by pixel analysis of the fused image to further determine the depth information of the mesh obstacles. This allows autonomous driving equipment to comprehensively perceive the surrounding environment based on the depth information of the mesh obstacles, achieving precise obstacle avoidance.

[0062] This invention acquires images to be identified in real time using a camera, then uses semantic segmentation technology to obtain semantic segmentation maps of the images to be identified, enabling accurate detection of mesh obstacles in real time. The depth map generated by binocular stereo imaging is then corrected and supplemented based on the set of semantic segmentation maps, improving the recognition accuracy of autonomous driving equipment for mesh obstacles. This allows robots or autonomous vehicles to have a comprehensive perception of the environment in complex environments, enabling precise obstacle avoidance and further improving driving safety.

[0063] In one embodiment of this application, the fully trained semantic segmentation prediction model is trained based on the PSPNet neural network;

[0064] The structure of the PSPNet neural network includes a feature extraction subnetwork, a pooling subnetwork, and a convolutional subnetwork.

[0065] First, it's important to clarify that PSPNet (Pyramid Scene Parsing Network), as a neural network model for mesh obstacle recognition, uses a pyramid pooling module as its core module. This module divides the feature layer into grids of different sizes, aggregating contextual information from different regions, thereby improving the ability to acquire global information. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is an overall framework diagram of a PSPNet provided in an embodiment of the present invention.

[0066] It should be noted that since the technical solutions in this invention are mostly used in embedded systems or other mobile platforms, the computational performance requirements cannot be too high, but a certain degree of accuracy is also required. After balancing performance and speed, the feature extraction sub-network in this invention selects the ResNet50 network. Using the backbone, one feature layer after another can be obtained as the input for the subsequent processing part.

[0067] Then regarding pooling subnetworks, please refer to [link / reference]. Figure 4 As shown in section c, features at four scales are fused. The top row is the coarsest global pooling, and the rows below are pooling at different scales. After a series of processing steps, upsampling is performed to restore the image to its initial size. Then, the images are stacked together to form the overall framework of PSPNet.

[0068] Understandably, through the above two steps, we have obtained the features of the input image. In order to obtain an image with the same dimensions as the input, we need to perform a final channel adjustment, that is, use a convolutional sub-network. For example, we use a 3x3 convolution to adjust the feature layer, use a 1x1 convolution to adjust the number of channels, and finally use a size adjustment method to adjust the image to be consistent with the input image, thus obtaining the final semantic segmentation map.

[0069] In one embodiment of this application, the training process of the semantic segmentation prediction model includes:

[0070] Obtain an image set including mesh obstacles, label the image set with classification tags, and obtain a classification result set;

[0071] The image set and the classification results corresponding to each image are combined to form a dataset, which includes a training set, a test set and a prediction set.

[0072] The training set is input into the PSPNet neural network for training. After the preset loss condition is reached, the trained model parameters are obtained, and the trained model parameters are loaded based on the PSPNet neural network to complete the training of the semantic segmentation prediction model.

[0073] Understandably, unmanned equipment, such as autonomous robots, is widely used in factory transportation, where there are many obstacles such as fences. Therefore, the image set of mesh obstacles in this invention mainly comes from factory environments. After taking a certain number of photos, the image set can be labeled with classification tags. Specifically, tagging software can be used to manually create tags, with the effect as follows: Figure 5 As shown, please refer to Figure 5 , Figure 5 This is an illustration of a label creation effect provided in one embodiment of the present invention. It can be understood that manually labeling images is equivalent to manually classifying the content in a set of images, thereby obtaining a dataset composed of images and their corresponding classification results, which can then be used for subsequent training.

[0074] It should be noted that transfer learning was used to accelerate training and improve prediction performance. In the pre-trained model, the output had 20 categories. To simplify the model, this training process adjusted the output classification to 5 categories: background, fence, ground, nylon netting, and people. This significantly accelerated the training speed and greatly improved subsequent prediction speed.

[0075] The training set is then input into the constructed PSPNet neural network for training. In a specific embodiment of the present invention,

[0076] The training set is input into the PSPNet neural network for training, specifically including:

[0077] The first image feature layer in the training set is extracted using a feature extraction subnetwork.

[0078] The first image feature layer is pooled at different scales using a pooling subnetwork to obtain the second image feature layer.

[0079] The number of feature layers and channels in the second image feature layer is adjusted using a convolutional subnetwork to make the output image the same size as the input image.

[0080] Understandably, after balancing performance and speed, this invention selects the ResNet50 network for the feature extraction sub-network. This network can be used to obtain the first image feature layer corresponding to the training set, which serves as the input for subsequent processing. Then, the pooling sub-network performs pooling operations at different scales on the first image feature layer to obtain the second image feature layer. The pooling sub-network can divide the first feature layer into grids of different sizes, then aggregate contextual information from different regions to obtain global information, finally generating the second image feature layer. The second image feature layer adds more detailed information compared to the first image feature layer, facilitating accurate recognition of semantic content in the image. Through these two steps, we have obtained the features of the input image. To obtain an image with the same dimensions as the input, a final channel adjustment is needed, i.e., using a convolutional sub-network. For example, a 3x3 convolution is used to adjust the feature layer, a 1x1 convolution is used to adjust the number of channels, and finally, a size adjustment method is used to adjust the image to match the input image, resulting in the final semantic segmentation map.

[0081] After reaching the preset training conditions, such as 100 iterations of training, the model file with the smallest combined loss on the training and test sets is selected from the model files generated in each generation to determine the neural network model parameters. Then, the trained model parameters are loaded into the constructed PSPNet model to complete the training of the semantic segmentation prediction model.

[0082] After training is complete, the prediction process begins. This process does not require backpropagation, and the neural network parameters do not need to be updated or learned. By inputting the original image into the fully trained semantic segmentation prediction model, a semantic segmentation image can be output.

[0083] In one embodiment of the present invention, after training the semantic segmentation prediction model is completed, the above method further includes:

[0084] The trained semantic segmentation prediction model is evaluated using preset evaluation metrics to obtain evaluation results.

[0085] Among the preset evaluation metrics, the mIoU metric is used for evaluation. mIoU is the ratio of the intersection and union of the two sets of true values ​​and predicted values. This invention evaluates the semantic segmentation prediction model trained above using mIoU, and the evaluation result is 76.03%, indicating that the model has a good prediction effect.

[0086] In one embodiment of this application, obtaining a depth map of the image to be identified includes:

[0087] Obtain the calibration parameters of the depth camera and correct the image to be recognized based on the calibration parameters;

[0088] The corrected image is matched, and the depth of each pixel in the image to be identified is calculated based on the matching results to obtain the depth map of the image to be identified.

[0089] It should be noted that there are currently three main depth camera technologies: structured light, binocular vision, and Time-of-Flight (TOF). This invention employs a binocular vision approach, meaning the depth camera used in this invention is a binocular camera. Binocular cameras mimic the principle of human eye ranging, utilizing the parallax of the images acquired by the left and right cameras to reconstruct the depth information of the image. Binocular cameras have strong resistance to strong light interference, can operate in outdoor environments, have the lowest manufacturing cost, and can be combined with deep learning to further optimize imaging.

[0090] The calibration parameters for a depth camera include the intrinsic and extrinsic parameters of the two cameras in a stereo camera setup, as well as the homography matrix between the two cameras. Camera calibration can be understood as a mapping from world coordinates to pixel coordinates. By obtaining the mapping relationship between world coordinates and pixel coordinates through calibration, the world coordinates can be deduced from the pixel coordinates of a pixel. The camera's intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length and pixel size. The camera's extrinsic parameters include the camera's parameters in the world coordinate system, such as the camera's position and rotation direction. The homography matrix between the two cameras describes the mapping relationship between two planes, that is, the transformation relationship of some points on the common plane between the two images.

[0091] Understandably, when calculating pixel depth, besides knowing the camera's focal length and baseline from its parameters, it's also necessary to know the disparity between the two cameras. This means knowing the correspondence between each pixel in the left camera and its corresponding point in the right camera. In other words, pixel matching is performed on the two calibrated images. Specifically, this can be done using the homography matrix of the two cameras or epipolar constraints. Once matching is complete, the disparity between the two cameras is obtained, allowing the calculation of the depth of each pixel, thus yielding a depth map of the image to be recognized. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a depth map of an image to be identified according to an embodiment of the present invention. The portion within the box represents the depth map of a mesh obstacle. As can be seen, the effect is not ideal. This is because the mesh targets are too small, and the mesh is often horizontally distributed, resulting in no significant parallax in the mesh seen by the left and right cameras of the binocular camera, making it difficult to identify the mesh obstacle.

[0092] In one embodiment of this application, image fusion of semantic segmentation map and depth map includes:

[0093] Construct a camera coordinate system based on the depth camera;

[0094] The semantic segmentation map and the depth map are fused in the camera coordinate system to obtain a fused image.

[0095] Understandably, since both the depth map and semantic segmentation map are obtained by further processing the image to be identified captured by a depth camera, and the processing does not involve a change in the coordinate system, the depth map and semantic segmentation map share the same coordinate system. Therefore, the camera coordinate system under the depth camera can be used to fuse the depth map and semantic segmentation map. Please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a fused image obtained by fusing a semantic segmentation map and a depth map of an image to be identified, as provided in an embodiment of the present invention.

[0096] In one embodiment of this application, pixel analysis is performed on the fused image to determine the depth information of the mesh obstacle, including:

[0097] Obtain the histogram corresponding to the fused image, perform pixel statistics on the histogram, and determine the depth information of the mesh obstacle based on the statistical results.

[0098] In the process of pixel analysis of the fused image, to improve analysis efficiency, the histogram corresponding to the region where the mesh obstacle is located can be obtained. Specifically, the region with mesh obstacles can be cropped from the fused image as the region of interest, and histogram pixel statistics can be performed. Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This invention provides a histogram corresponding to a portion of a fused image region as an embodiment of the present invention.

[0099] The histogram shows that pixel values ​​are mainly distributed around two peaks. The region with peak P1 has the highest pixel value, which is easily identified as the closest distance to the obstacle, i.e., the distance to the mesh obstacle. The region with peak P2 and lower pixel values ​​represents other objects behind the mesh. Therefore, the depth information of the mesh obstacle can be recalculated based on the pixel values ​​of the region with peak P1.

[0100] In one embodiment of this application, the method further includes:

[0101] The depth map is filled and repaired based on depth information.

[0102] It is understandable that after obtaining the depth information of the mesh obstacle, the depth map can be filled and repaired based on the depth information of the mesh obstacle, specifically by replacing it with the recalculated depth information of the mesh obstacle.

[0103] Based on the depth information of the mesh obstacles generated by binocular stereo imaging, the depth map is corrected and supplemented, enabling robots or autonomous vehicles to fully perceive the environment in complex environments and achieve precise obstacle avoidance.

[0104] To better implement the mesh obstacle recognition method in this invention embodiment, based on the mesh obstacle recognition method, please refer to the corresponding documentation. Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the mesh obstacle recognition device provided by the present invention. The present invention also provides a second aspect, and a mesh obstacle recognition device 900, comprising:

[0105] The first acquisition module 901 is used to acquire an image to be identified, wherein the image to be identified includes a mesh obstacle;

[0106] The output module 902 is used to input the image to be recognized into a fully trained semantic segmentation prediction model and output a semantic segmentation map of the image to be recognized.

[0107] The second acquisition module 903 is used to acquire the depth map of the image to be identified;

[0108] The determination module 904 is used to perform image fusion of the semantic segmentation map and the depth map, and to perform pixel analysis on the fused image to determine the depth information of the mesh obstacle.

[0109] It should be noted that the device 900 provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding content in the above method embodiments, and will not be repeated here.

[0110] Based on the above-described mesh obstacle recognition method, this invention also provides an electronic device, including: a processor and a memory, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, it implements the steps of the mesh obstacle recognition method as described in the above embodiments.

[0111] Figure 10 The diagram shows a structural schematic of an electronic device 1000 suitable for implementing embodiments of the present invention. The electronic device in the embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0112] The electronic device includes a memory and a processor, wherein the processor may be referred to as processing device 1001 below, and the memory may include at least one of read-only memory (ROM) 1002, random access memory (RAM) 1003 and storage device 1008 below, as detailed below:

[0113] like Figure 10 As shown, the electronic device 1000 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0114] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0115] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, it performs the functions defined in the methods of the embodiments of the present invention.

[0116] Based on the above-described mesh obstacle recognition method, this embodiment of the invention also provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the mesh obstacle recognition method of the above embodiments.

[0117] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0118] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying mesh obstacles, characterized in that, include: Acquire an image to be identified, wherein the image to be identified includes a mesh of obstacles; The image to be identified is input into a fully trained semantic segmentation prediction model, which outputs a semantic segmentation map of the image to be identified. Obtain the depth map of the image to be identified; The semantic segmentation map and the depth map are fused together, and pixel analysis is performed on the fused image to determine the depth information of the mesh obstacles. The step of fusing the semantic segmentation map with the depth map includes: Construct a camera coordinate system based on the depth camera; The semantic segmentation map and the depth map are fused in the camera coordinate system to obtain a fused image. The pixel analysis of the fused image to determine the depth information of the mesh obstacle includes: Obtain the histogram corresponding to the fused image, perform pixel statistics on the histogram, and determine the depth information of the mesh obstacle based on the statistical results; It also includes: filling and patching the depth map based on the depth information.

2. The mesh obstacle recognition method according to claim 1, characterized in that, The fully trained semantic segmentation prediction model is trained based on the PSPNet neural network; The structure of the PSPNet neural network includes a feature extraction subnetwork, a pooling subnetwork, and a convolutional subnetwork.

3. The mesh obstacle recognition method according to claim 2, characterized in that, The training process of the semantic segmentation prediction model includes: Obtain an image set including mesh obstacles, and label the image set with classification tags to obtain a classification result set; The image set and the classification results corresponding to each image are combined to form a dataset, wherein the dataset includes a training set, a test set and a prediction set; The training set is input into the PSPNet neural network for training. After the preset loss condition is reached, the trained model parameters are obtained, and the trained model parameters are loaded based on the PSPNet neural network to complete the training of the semantic segmentation prediction model. Specifically, inputting the training set into the PSPNet neural network for training includes: The feature extraction subnetwork is used to extract the first image feature layer from the training set; The pooling subnetwork is used to perform pooling operations at different scales on the first image feature layer to obtain the second image feature layer. The number of feature layers and channels of the second image feature layer are adjusted using the convolutional subnetwork to make the output image the same size as the input image.

4. The mesh obstacle recognition method according to claim 1, characterized in that, The process of obtaining the depth map of the image to be identified includes: Obtain the calibration parameters of the depth camera, and correct the image to be identified based on the calibration parameters; The corrected image is matched, and the depth of each pixel in the image to be identified is calculated based on the matching results to obtain the depth map of the image to be identified.

5. A mesh obstacle recognition device, characterized in that, include: The first acquisition module is used to acquire an image to be identified, wherein the image to be identified includes a mesh obstacle; The output module is used to input the image to be recognized into a fully trained semantic segmentation prediction model and output a semantic segmentation map of the image to be recognized. The second acquisition module is used to acquire the depth map of the image to be identified; The determination module is used to perform image fusion of the semantic segmentation map and the depth map, and to perform pixel analysis on the fused image to determine the depth information of the mesh obstacle; The step of fusing the semantic segmentation map with the depth map includes: Construct a camera coordinate system based on the depth camera; The semantic segmentation map and the depth map are fused in the camera coordinate system to obtain a fused image. The pixel analysis of the fused image to determine the depth information of the mesh obstacle includes: Obtain the histogram corresponding to the fused image, perform pixel statistics on the histogram, and determine the depth information of the mesh obstacle based on the statistical results; It also includes: filling and patching the depth map based on the depth information.

6. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory is used to store a program; and the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the mesh obstacle recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, are capable of implementing the steps in the mesh obstacle recognition method according to any one of claims 1 to 4.

Images