Image recognition model training method, image recognition method, and storage medium

Abstract

The application relates to the technical field of image processing, and discloses an image recognition model training method, an image recognition method and a storage medium, the image recognition model training method comprising the following steps: acquiring point cloud data and an RGB image labeled with semantic information; projecting the point cloud data into an image coordinate system to obtain a projection image; inputting the RGB image and the projection image into first and second feature extraction units respectively to obtain first and second feature maps of multiple scales; performing feature fusion on the first and second feature maps of the same scale through a feature fusion module to obtain fusion feature maps of multiple scales; identifying the RGB image according to the multiple fusion feature maps through a decoder to obtain a first vector; generating an identification result based on the first vector through an identification result generation module; and adjusting the parameters of the image recognition model based on the identification result and the semantic information through a parameter optimization module. The application improves the accuracy of the image recognition result.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular to an image recognition model training method, an image recognition method and a storage medium. BACKGROUND

[0002] Three-dimensional reconstruction is a technology of reconstructing three-dimensional structure of an object or a scene by analyzing and processing two-dimensional images or scanning data taken from different perspectives. With the development of computer vision technology, three-dimensional reconstruction is widely applied in the fields of virtual reality (VR), augmented reality (AR) and the like.

[0003] In a three-dimensional reconstruction technology based on vision, in order to construct a virtual three-dimensional scene highly similar to a real three-dimensional scene, the real three-dimensional scene can be first photographed from multiple angles to obtain two-dimensional images, and then object recognition is performed on the obtained images to obtain an image recognition result, which includes objects included in the images. Therefore, the objects included in the real three-dimensional scene can be determined according to the image recognition result, and then three-dimensional reconstruction can be performed based on the image recognition result. However, if the accuracy of the image recognition result is low, the three-dimensional scene cannot be accurately reconstructed. How to improve the accuracy of the image recognition result is a problem to be solved. SUMMARY

[0004] In view of the above problems, embodiments of the present application provide an image recognition model training method, an image recognition method, an electronic device and a storage medium, to solve the problem of low accuracy of the image recognition result in the prior art.

[0005] According to an aspect of the embodiments of the present application, a method for training an image recognition model is provided. The image recognition model includes a feature extraction module, a feature fusion module, a decoder, an identification result generation module, and a parameter optimization module. The image recognition model is used to identify a plurality of objects. The method includes: obtaining an RGB image, wherein the RGB image is an image taken by a target region in a real three-dimensional scene, and each pixel point in the RGB image is labeled with semantic information; obtaining point cloud data collected from the target region in the real three-dimensional scene; projecting the point cloud data into an image coordinate system to obtain a projection image; inputting the RGB image into a first feature extraction unit in the feature extraction module to perform feature extraction on the RGB image and obtain a first feature map set, wherein the first feature map set includes a plurality of first feature maps of different scales; inputting the projection image into a second feature extraction unit in the feature extraction module to perform feature extraction on the projection image and obtain a second feature map set, wherein the second feature map set includes a second feature map of each scale in the plurality of scales; performing feature fusion on first feature maps and second feature maps of the same scale in the first feature map set and the second feature map set by the feature fusion module to obtain a plurality of fusion feature maps corresponding to the plurality of scales; performing identification and prediction on the RGB image according to the plurality of fusion feature maps by the decoder to obtain a first vector, wherein the first vector includes a prediction score of each pixel point in the RGB image belonging to each object in the plurality of objects; generating an identification result based on the first vector by the identification result generation module, wherein the identification result includes an object category to which each pixel point in the RGB image belongs; and adjusting parameters of the image recognition model based on the identification result and the semantic information by the parameter optimization module.

[0006] In one optional manner, the plurality of fused feature maps includes a first-scale fused feature map, a second-scale fused feature map, a third-scale fused feature map, and a fourth-scale fused feature map; the step of recognizing and predicting the RGB image based on the plurality of fused feature maps by the decoder to obtain a first vector includes: upsampling the first-scale fused feature map to obtain a second-scale third feature map; fusing the second-scale fused feature map and the second-scale third feature map to obtain a second-scale fourth feature map; upsampling the second-scale fourth feature map to obtain a third-scale fifth feature map; fusing the third-scale fused feature map and the third-scale fifth feature map to obtain a third-scale sixth feature map; upsampling the third-scale sixth feature map to obtain a fourth-scale seventh feature map; fusing the fourth-scale fused feature map and the fourth-scale seventh feature map to obtain a fourth-scale eighth feature map; and recognizing and predicting the RGB image based on the fourth-scale eighth feature map to obtain the first vector.

[0007] In an optional embodiment, the image recognition model further includes an edge feature optimization module, which comprises a first unit, a second unit, and a residual fusion module. The first unit includes an activation function layer, three convolutional layers, and an edge feature determination layer. The second unit includes a confidence determination layer and an adaptive gating layer. Before generating a recognition result based on the first vector through the recognition result generation module, the method further includes: processing the first vector through the activation function layer in the first unit to obtain a second vector; extracting edge features from the second vector through the three convolutional layers to obtain a third vector; determining edge features based on the third vector through the edge feature determination layer to obtain an edge vector; determining the confidence of the first vector through the confidence determination layer; outputting a gating signal to the residual fusion module based on the confidence of the first vector through the adaptive gating layer; and performing residual fusion processing based on the gating signal and the edge vector and the first vector through the residual fusion module to obtain a processed vector. Generating a recognition result based on the first vector through the recognition result generation module includes: generating the recognition result based on the processed vector through the recognition result generation module.

[0008] In an optional manner, the first unit further comprises a morphological dilation layer; before the residual fusion module performs residual fusion processing on the edge vector and the first vector according to the gating signal to obtain the processed vector, the method further comprises: performing dilation processing on the edge vector by the morphological dilation layer to obtain a dilated edge vector; the residual fusion module performs residual fusion processing on the dilated edge vector and the first vector according to the gating signal to obtain the processed vector; or the first unit further comprises a morphological erosion layer, before the residual fusion module performs residual fusion processing on the edge vector and the first vector according to the gating signal to obtain the processed vector, the method further comprises: performing erosion processing on the edge vector by the morphological erosion layer to obtain an eroded edge vector; the residual fusion module performs residual fusion processing on the eroded edge vector and the first vector according to the gating signal to obtain the processed vector.

[0009] In an optional manner, the parameter optimization module adjusts the parameters of the image recognition model based on the recognition result and the semantic information, comprising: the parameter optimization module determines a basic loss value according to the recognition result and the semantic information; determines a connectivity loss value according to the first vector; determines an edge consistency loss value based on the processed vector; determines a total loss value according to the basic loss value, the connectivity loss value and the edge consistency loss value; adjusts the parameters of the image recognition model according to the total loss value.

[0010] In an optional manner, after the RGB image is obtained, the method further comprises: adding a line on the RGB image, which is adapted to the shape of an object in the plurality of objects, and labeling semantic information for the pixel point corresponding to the line to update the RGB image; the RGB image is input into the first feature extraction unit in the feature extraction module, comprising: inputting the updated RGB image into the first feature extraction unit in the feature extraction module.

[0011] According to another aspect of the embodiments of the present application, an image recognition method is provided. The method comprises: obtaining an image to be recognized; dividing the image to be recognized into a plurality of sub-images to be recognized, wherein adjacent sub-images to be recognized have overlapping regions; inputting the plurality of sub-images to be recognized into a trained image recognition model respectively to obtain a recognition result corresponding to each of the sub-images to be recognized output by the trained image recognition model, wherein the trained image recognition model is trained by the image recognition model training method described above; and determining a recognition result of the image to be recognized according to the recognition result corresponding to each of the sub-images to be recognized.

[0012] In an optional manner, the determining of the recognition result of the image to be recognized according to the recognition result corresponding to each of the sub-images to be recognized comprises: taking the recognition result corresponding to a non-overlapping region in each of the sub-images to be recognized as the recognition result of a region in the image to be recognized that is the same as the non-overlapping region; and for an overlapping region in each of the sub-images to be recognized, determining a recognition result corresponding to the overlapping region from recognition results corresponding to all sub-images to be recognized containing the overlapping region, obtaining a plurality of recognition results, performing weighted fusion processing on the plurality of recognition results to obtain a final recognition result corresponding to the overlapping region, and taking the final recognition result as the recognition result of a region in the image to be recognized that is the same as the overlapping region.

[0013] According to another aspect of the embodiments of the present application, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the image recognition model training method described above and / or the image recognition method described above.

[0014] According to yet another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the image recognition model training method described above and / or the image recognition method described above.

[0015] In the embodiment of the present application, the point cloud data is projected into the image coordinate system to extract the accurate spatial and structural information contained therein, obtain a projection image, and perform feature extraction on the projection image and the RGB image respectively to obtain feature maps. Then, the feature maps are fused to make the obtained fused feature map have both the rich texture details of the RGB image and the accurate three-dimensional geometric information of the point cloud data, so that the image recognition model can utilize both two-dimensional visual features and three-dimensional geometric constraints in the process of learning the fused feature map, thereby more accurately distinguishing independent objects that are mutually adhered in the RGB image. In general, through the supervised training of the fused RGB image and point cloud data, the accuracy of the trained image recognition model is improved, so that it can output more accurate image recognition results in subsequent identification of other images.

[0016] The above description is only a summary of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, the embodiments of the present application can be implemented in accordance with the content of the description, and in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the application and, together with the description, serve to explain the principles of the application. In the drawings:

[0018] Figure 1 A flowchart of an image recognition model training method provided by an embodiment of the present application is shown;

[0019] Figure 2 A structure diagram of an image recognition model provided by an embodiment of the present application is shown;

[0020] Figure 3 A principle diagram for determining a fused feature map provided by an embodiment of the present application is shown;

[0021] Figure 4 A sub-step flowchart of step 170 provided by an embodiment of the present application is shown;

[0022] Figure 5 A principle diagram for determining a first vector provided by an embodiment of the present application is shown;

[0023] Figure 6 A structure diagram of an edge feature optimization module provided by an embodiment of the present application is shown;

[0024] Figure 7 A structure diagram of an edge feature optimization module provided by another embodiment of the present application is shown;

[0025] Figure 8A flowchart of an image recognition method provided by an embodiment of the present application is shown.

[0026] Figure 9 A schematic diagram of a to-be-recognized image and multiple to-be-recognized sub-images provided by an embodiment of the present application is shown.

[0027] Figure 10 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0029] In the process of three-dimensional reconstruction, the real three-dimensional scene needs to be photographed from multiple angles to obtain multiple images, and the multiple images are respectively identified to obtain image recognition results, and then three-dimensional reconstruction can be performed based on the image recognition results.

[0030] Since the three-dimensional scene to be reconstructed is usually large, the number of images obtained after the real three-dimensional scene is photographed from multiple angles is large. In order to improve the efficiency of obtaining image recognition results, an image recognition model is usually used to identify images. By inputting a to-be-identified image into the image recognition model, the image recognition result output by the image recognition model can be obtained. Therefore, the accuracy of the image recognition result output by the image recognition model is related to the precision of the image recognition model. The higher the precision of the image recognition model is, the higher the accuracy of the image recognition result output by the image recognition model is, and thus the accuracy of the obtained image recognition result can be improved by improving the precision of the image recognition model.

[0031] The precision of the image recognition model can be improved by supervised training of the image recognition model. Specifically, when training the image recognition model, the objects in the training images are first labeled by category, and then the labeled training images are input into the image recognition model, so that the model establishes a mapping relationship between the visual features of the images and the corresponding category labels through iterative learning, masters the image recognition capability, and thus completes the training of the model.

[0032] However, when there are multiple objects in the real three-dimensional scene, especially objects with adjacent spatial positions or objects in contact, in the two-dimensional images obtained by photographing, these objects will form adjacent or even partially overlapping image regions on the image. In this case, even if the training images contain category labels of the objects, the model may still misjudge two independent objects as a whole when identifying and learning the training images, resulting in poor training effect of the model, and thus a high-precision image recognition model cannot be obtained.

[0033] Three-dimensional data refers to data with three dimensions. Three-dimensional data includes point cloud data, mesh data, and voxel data, etc. Point cloud data is composed of a series of points in a three-dimensional space, each point has X-axis, Y-axis, and Z-axis coordinate information, and can also include other attribute information such as color, etc. Point cloud data is often used to represent the outer surface of an object. For example, after scanning a real three-dimensional scene by a laser scanner, a three-dimensional model corresponding to the shape of the object in the real three-dimensional scene composed of point cloud can be obtained. Therefore, for two objects in a real three-dimensional scene that are adjacent in space or in contact, the point cloud data can retain the real spatial positions and topological relationships of each object in the real three-dimensional scene, and fundamentally overcome the occlusion and adhesion problems caused by projection in two-dimensional images. Even if two objects overlap in a two-dimensional view, they are still clearly separated objects in a three-dimensional coordinate system in the point cloud data.

[0034] Based on this, when training an image recognition model, if the model can be trained using point cloud data at the same time, the model can be provided with depth and spatial structure information beyond two-dimensional images, so that the model can effectively distinguish objects that are adhesively connected or occluded in two-dimensional images, thereby improving the training effect of the model and ultimately obtaining an image recognition model with high precision.

[0035] Figure 1 A flowchart of an image recognition model training method provided by an embodiment of the present application is shown. The method is executed by an electronic device, which can be a device including one or more processors, such as a tablet computer, a computer, or a server. The processor can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present application, without limitation. The one or more processors included in the electronic device can be the same type of processor, such as one or more CPUs; or different types of processors, such as one or more CPUs and one or more ASICs, without limitation. The method is used to train an image recognition model of a feature extraction module, a feature fusion module, a decoder, a recognition result generation module, and a parameter optimization module. Figure 2 A structural diagram of an image recognition model provided by an embodiment of the present application is shown. Among them, Figure 1 The image recognition model training method provided is used to train Figure 2 The image recognition model provided. As Figure 1 shown, the image recognition model training method includes the following steps 110-190.

[0036] Step 110: Obtain an RGB image.

[0037] If it is necessary to reconstruct a virtual three-dimensional scene corresponding to a certain region (i.e., a target region) in a real three-dimensional scene, the target region can be photographed by using a camera, so as to obtain an RGB image. It can be understood that if the target region is large or the target region includes many objects, the target region can be photographed from multiple different angles to obtain multiple RGB images, so as to ensure that the obtained RGB image includes complete information of the scene of the target region and reduce the data blind area caused by the occlusion of objects in the target region.

[0038] It should be noted that if the target region is large, the target region can be divided into multiple target sub-regions, and then the camera is used to photograph each target sub-region, so as to improve the spatial resolution and detail coverage of the obtained RGB image.

[0039] In order to train the image recognition model by using the RGB image, each pixel point in the RGB image obtained in this step is labeled with semantic information of an object category to which the pixel point belongs. For example, if an object category to which a certain pixel point in an RGB image belongs is a cat, the semantic information labeled by the pixel point is a cat.

[0040] It should be noted that if the image recognition model needs to recognize multiple objects, only the pixel points in the RGB image belonging to each object in the multiple objects are labeled. For example, if the image recognition model needs to recognize cats and dogs, only the pixel points in the RGB image belonging to the categories of cats and dogs are labeled, and other pixel points in the RGB image belonging to objects other than cats and dogs do not need to be labeled. In this way, when the image recognition model is trained by using the RGB image, the image learning model can focus on learning the core features of the target category, rather than being disturbed by irrelevant background information, so as to realize accurate recognition of specific objects in a more efficient and robust manner.

[0041] Step 120: Obtain point cloud data collected from a target region in a real three-dimensional scene.

[0042] The target region in the real three-dimensional scene can be scanned by using a laser radar, so as to obtain a three-dimensional model corresponding to the shape of the object in the real three-dimensional scene, which is composed of point clouds.

[0043] It should be noted that the point cloud data obtained in this step corresponds to the same region in the real three-dimensional scene as the RGB image obtained in step 110, that is, the objects included in the RGB image are the same as the objects included in the point cloud data.

[0044] Step 130: Project the point cloud data into an image coordinate system to obtain a projection image.

[0045] Specifically, point cloud data can be transformed from the LiDAR coordinate system to the camera coordinate system based on the extrinsic parameters (rotation and translation matrices) between the LiDAR that collects point cloud data and the camera that captures the target area, thereby projecting the point cloud data onto the image coordinate system to obtain a projected image.

[0046] Step 140: Input the RGB image into the first feature extraction unit in the feature extraction module to extract features from the RGB image and obtain the first feature map set.

[0047] Figure 3 A schematic diagram illustrating the principle of determining the fusion feature map provided in an embodiment of this application is shown. For example... Figure 3 As shown, an RGB image is input into a first feature extraction unit, which extracts features from the RGB image to obtain a first feature map at a first scale, a first feature map at a second scale, a first feature map at a third scale, and a first feature map at a fourth scale. These first feature maps at the first, second, third, and fourth scales constitute a first feature map set. The first, second, third, and fourth scales can be set as needed; for example, the first to fourth scales can be 1 / 32, 1 / 16, 1 / 8, and 1 / 4 scales of the RGB image, respectively. The following description uses an example where the first to fourth scales are 1 / 32, 1 / 16, 1 / 8, and 1 / 4 scales of the RGB image, respectively.

[0048] It is worth noting that in this embodiment, only four scales are used as an example. In some embodiments, only two scales of the first feature map or five or more scales of feature maps may be determined.

[0049] Step 150: Input the projected image into the second feature extraction unit in the feature extraction module to extract features from the projected image and obtain the second feature map set.

[0050] like Figure 3 As shown, the projected image is input into the second feature extraction unit, which extracts progress features from the projected image to obtain a second feature map at a first scale, a second feature map at a second scale, a second feature map at a third scale, and a second feature map at a fourth scale. The second feature map at the first scale, the second feature map at the second scale, the second feature map at the third scale, and the second feature map at the fourth scale constitute the second feature map set.

[0051] Step 160: The feature fusion module performs feature fusion on the first feature map and the second feature map with the same scale in the first feature map set and the second feature map set to obtain multiple fused feature maps corresponding to multiple scales.

[0052] As shown in Figure 2 and Figure 3 After the first feature map set and the second feature map set are input into the feature fusion module, the feature fusion module fuses the first feature map of the first scale and the second feature map of the first scale to obtain a first scale fusion feature map; fuses the first feature map of the second scale and the second feature map of the second scale to obtain a second scale fusion feature map; fuses the first feature map of the third scale and the second feature map of the third scale to obtain a third scale fusion feature map; and fuses the first feature map of the fourth scale and the second feature map of the fourth scale to obtain a fourth scale fusion feature map.

[0053] Step 170: The RGB image is recognized and predicted by the decoder according to the plurality of fusion feature maps to obtain a first vector.

[0054] If the image recognition model provided by Figure 2 is used to recognize a plurality of objects, in this step, after the first scale fusion feature map, the second scale fusion feature map, the third scale fusion feature map and the fourth scale fusion feature map are input into the decoder, the decoder recognizes and predicts the fusion feature maps of multiple scales to obtain the prediction scores of each pixel point in the RGB image in step 110 belonging to various objects in the plurality of objects. For example, if the image recognition model provided by Figure 2 is used to recognize cats and dogs, the first vector obtained in this step includes the prediction scores of each pixel point belonging to a cat and belonging to a dog. The higher the prediction score of a certain pixel point belonging to a certain object category, the greater the probability that the pixel point belongs to the object of the category.

[0055] Step 180: An identification result is generated by the identification result generation module based on the first vector.

[0056] The first vector output by the decoder is the prediction score of each pixel point belonging to various objects in the plurality of objects, and the prediction score is an unnormalized score. Therefore, in this step, the first vector can be normalized by the Softmax function to obtain the probability distribution of each pixel point in various object categories, and then the class index with the highest probability (Argmax) is taken for each pixel point to directly generate the final multi-class segmentation map, thereby obtaining the identification result. The identification result includes the object category to which each pixel point in the RGB image belongs.

[0057] Step 190: The parameters of the image recognition model are adjusted by the parameter optimization module based on the identification result and the semantic information.

[0058] Since the recognition result is obtained by the image recognition model identifying and predicting the RGB image and the projection image, and the semantic information labeled by the RGB image is real information, in this step, the loss value can be determined based on the recognition result and the semantic information, and the parameters of the image recognition model can be adjusted according to the loss value, for example, the weights and biases of the image recognition model are adjusted.

[0059] In the embodiments of the present application, the point cloud data is projected into the image coordinate system to extract the accurate spatial and structural information contained therein, and a projection image is obtained. Then, feature extraction is performed on the projection image and the RGB image respectively, and a feature map is obtained. After that, the feature maps are fused to make the obtained fused feature map have both the rich texture details of the RGB image and the accurate three-dimensional geometric information of the point cloud data, so that the image recognition model can utilize both two-dimensional visual features and three-dimensional geometric constraints in the process of learning the fused feature map, thereby more accurately distinguishing independent objects that are mutually adhered in the RGB image. In general, through the supervised training of fusing the RGB image and the point cloud data, the accuracy of the trained image recognition model is improved, so that it can output more accurate image recognition results in subsequent identification of other images.

[0060] In order to improve the accuracy of the determined first vector, Figure 4 A sub-step flowchart of step 170 provided by the embodiments of the present application is shown. As shown in Figure 4 Step 170 includes steps 171-177.

[0061] Step 171: performing up-sampling processing on the first scale fused feature map to obtain a second scale third feature map.

[0062] In order to better introduce how to determine the first vector, Figure 5 A principle diagram for determining the first vector provided by the embodiments of the present application is shown. As shown in Figure 5 In this step, the first scale feature fusion map is up-sampled to obtain a second scale third feature map.

[0063] Step 172: performing fusion processing on the second scale fused feature map and the second scale third feature map to obtain a second scale fourth feature map.

[0064] As shown in Figure 5 In this step, the third feature map and the second scale fused feature map of the same scale are fused to obtain a fourth feature map.

[0065] Step 173: performing up-sampling processing on the second scale fourth feature map to obtain a third scale fifth feature map.

[0066] As shown in Figure 5As shown, in order to fuse the fourth feature map with the third-scale fused feature map, the fourth feature map at the second scale is upsampled in this step, and a fifth feature map with the same scale as the third-scale fused feature map is generated.

[0067] Step 174: Perform fusion processing on the third-scale fused feature map and the third-scale fifth feature map to obtain the third-scale sixth feature map.

[0068] like Figure 5 As shown, in this step, the fifth feature map and the third scale fusion feature map of the same scale are fused to obtain the sixth feature map.

[0069] Step 175: Upsample the sixth feature map at the third scale to obtain the seventh feature map at the fourth scale.

[0070] like Figure 5 As shown, in order to fuse the sixth feature map with the fourth-scale fusion feature map, the sixth feature map at the third scale is upsampled in this step, and the seventh feature map has the same scale as the fourth-scale fusion feature map.

[0071] Step 176: Perform fusion processing on the fourth-scale fused feature map and the fourth-scale seventh feature map to obtain the fourth-scale eighth feature map.

[0072] like Figure 5 As shown, in this step, the seventh feature map and the fourth scale fusion feature map of the same scale are fused to obtain the eighth feature map.

[0073] Step 177: Recognize and predict the RGB image based on the fourth scale eighth feature map to obtain the first vector.

[0074] This step is similar to step 170, so the principle and implementation method of this step can be referred to step 170, and will not be repeated here.

[0075] Large-scale feature maps are similar in size to the original image and have high spatial resolution, thus preserving rich spatial details. Because the downsampling of the original image is relatively shallow, the pixels in the resulting large-scale feature map correspond quite accurately to those in the original image, allowing for a clear depiction of object edges, contours, and textures. However, this focus on local details also has limitations. Each pixel in the large-scale feature map has a smaller receptive field, lacking a grasp of global information. Therefore, if an image recognition model only learns from large-scale feature maps, it may struggle to understand the overall concept of objects and the macroscopic layout and semantics of the scene in the original image.

[0076] The small-scale feature map is obtained by multiple down-sampling of the original image, and has a low spatial resolution, which does not retain the accurate position and edge details of the object and loses most of the spatial positioning information. However, each pixel point of the small-scale feature map can integrate a large range of information in the original image, and thus contains rich high-level semantics. Therefore, the image recognition model can understand the overall concept of the object and the macro layout of the scene in the original image by learning the small-scale feature map.

[0077] Therefore, in Figure 1 Based on the embodiments provided, for Figure 4 According to the embodiments provided, by fusing the large-scale feature map and the small-scale feature map, since the fusion process includes up-sampling the small-scale feature map to match the size of the large-scale feature map, and then fusing the feature maps of the same scale by splicing or element-by-element addition, etc., finally obtaining the fourth-scale eighth feature map, the fourth-scale eighth feature map integrates the high-resolution spatial details of the large-scale feature map and the rich high-level semantics of the small-scale feature map, that is, the fourth-scale eighth feature map not only retains the accurate positioning ability, but also has strong global understanding ability. Therefore, when the subsequent image recognition model performs recognition and prediction according to the fourth-scale eighth feature map, the image recognition model can accurately recognize the object category according to the global semantics from the small-scale feature map, effectively avoiding misjudgment; and accurately position the object boundary according to the fine spatial details retained by the large-scale feature map. In the above manner, the first vector generated by the image recognition model is not only more accurate in semantic category, but also more consistent with the real situation in spatial geometric structure, thereby improving the accuracy of the first vector determined by the image recognition model.

[0078] In order to further improve the accuracy of the image recognition model, in some embodiments, the image recognition model further includes an edge feature optimization module. Figure 6 A structural schematic diagram of the edge feature optimization module provided by the embodiments of the present application is shown. As Figure 6 shown, the edge feature optimization module includes a first unit, a second unit and a residual fusion module, the first unit includes an activation function layer, three convolution layers, an edge feature determination layer, the second unit includes a confidence determination layer and an adaptive gating layer.

[0079] Before step 180, the image recognition model training method further includes steps a1-a6.

[0080] Step a1: processing the first vector by the activation function layer in the first unit to obtain a second vector.

[0081] In this step, the first vector is normalized by an activation function (Softmax) layer to obtain a second vector, so that the gradient reflects the semantic boundary change.

[0082] Step a2: edge feature extraction is performed on the second vector by a three-layer convolutional layer to obtain a third vector.

[0083] In this step, the edge position of the object is extracted by a three-layer learnable CNN network, so that the high response area of the output corresponds to the predicted object boundary.

[0084] Step a3: edge features are determined from the third vector by an edge feature determination layer to obtain an edge vector.

[0085] In this step, the edge vector can be obtained by edge thresholding processing on the second vector. Specifically, an edge threshold can be pre-set, and the element values in the third vector that are greater than the edge threshold are retained, while the element values that are less than or equal to the edge threshold are set to zero (or a background value). The processed vector is the edge vector. In this way, weak response noise can be suppressed and significant boundary regions can be retained.

[0086] Step a4: the confidence of the first vector is determined by a confidence determination layer.

[0087] In this step, the confidence of the first vector is determined so that subsequent residual fusion of the edge vector and the first vector can be based on the confidence. Specifically, for the prediction score of a certain pixel point in the first vector, if the probability of the pixel point belonging to a certain object category is much higher than other categories, it indicates that the prediction result of the image recognition model has a high degree of credibility, and accordingly the confidence of the first vector is determined to be a higher confidence. Conversely, if the prediction scores of multiple categories are similar, it indicates that the image recognition model has ambiguity in judgment, and accordingly the confidence of the first vector is determined to be a lower confidence.

[0088] Step a5: an adaptive gating layer outputs a gating signal to the residual fusion module according to the confidence of the first vector.

[0089] The gating signal is used to indicate whether the residual fusion processing needs to be performed on the first vector and the edge vector. The gating signal includes a first gating signal and a second gating signal. If the confidence of the first vector is less than the confidence threshold, it indicates that the confidence of the first vector is low, and the first gating signal is output to the residual fusion module, so that the residual fusion module performs residual fusion on the first vector and the edge vector to improve the accuracy of the fused vector. If the confidence of the first vector is greater than or equal to the confidence threshold, it indicates that the confidence of the first vector is high, and the second gating signal is output to the residual fusion module, so that the residual fusion module does not need to perform residual fusion on the first vector and the edge vector.

[0090] Step a6: performing residual fusion processing on the edge vector and the first vector by the residual fusion module based on the gating signal to obtain a processed vector.

[0091] If the gating signal received by the residual fusion module is the first gating signal, the residual fusion module performs residual fusion processing on the first vector and the edge vector, thereby obtaining and outputting a processed vector (i.e., a residual fused vector). If the gating signal received by the residual fusion module is the second gating signal, the residual fusion module does not perform residual fusion on the first vector and the edge vector, and the residual fusion module outputs the first vector, that is, the processed vector is the first vector.

[0092] After performing the above steps a1 to a6, in order to obtain a recognition result, step 180 includes: generating a recognition result based on the processed vector by the recognition result generation module.

[0093] For the image recognition model including the edge feature optimization module, the edge feature optimization module determines an edge vector from the first vector, and then determines whether to perform residual fusion on the first vector and the edge vector based on the confidence of the first vector through the adaptive gating layer. When the model determines that the confidence of the first vector is low, the first vector is corrected and optimized by introducing more accurate edge vector information, thereby optimizing the boundary details. When the model determines that the confidence of the first vector is high, the first vector is retained to avoid unnecessary interference. This way of determining whether to correct and optimize the first vector according to the confidence of the first vector makes the model not only effectively handle difficult cases such as object boundary blur and adhesion, but also maintain efficient processing of simple scenes, further improving the accuracy of the image recognition model, so that a more accurate and robust recognition result can be obtained.

[0094] In the process of three-dimensional reconstruction, if a virtual object corresponding to an object in the real three-dimensional scene is not required to be included in the virtual three-dimensional scene obtained by reconstruction, all regions of the object in the RGB image need to be identified and deleted, and then three-dimensional reconstruction is performed on the RGB image without the object to obtain a virtual three-dimensional scene without the object. For example, in order to perform three-dimensional reconstruction on the sky, the sky needs to be photographed. If there is a power line in the sky, the power line will also be included in the RGB image obtained by photographing. If it is required that the virtual sky obtained by reconstruction does not include the power line, the first pixel point belonging to the power line in the image needs to be identified and deleted, and then three-dimensional reconstruction is performed on the RGB image without the power line to obtain a virtual sky without the power line. However, if all pixel points belonging to the power line cannot be identified in the RGB image, these pixel points belonging to the power line that are not deleted will be retained in the process of three-dimensional reconstruction, and finally the virtual sky obtained by reconstruction will have a power line remaining, which will seriously affect the visual quality of the reconstruction.

[0095] In order to avoid the above problems, a morphological dilation layer can be set in the image recognition model. Morphological dilation is a basic image processing technique that uses a probe called "structure element" to expand the target region in the image. For example, in the object recognition task, once the model identifies that a certain region belongs to a specific object, the morphological dilation layer will also include the pixel points within a certain range adjacent to the region into the object, thereby realizing the expansion or filling of the target region. For example, if the model identifies that the 5th to 15th pixel points in the first row of the RGB image belong to the power line, the morphological dilation layer will regard the 3rd to 4th pixel points and the 16th to 18th pixel points in the row as pixel points of the power line.

[0096] Therefore, in some embodiments, the first unit further includes a morphological dilation layer. Figure 7 The structure of the edge feature optimization module provided by another embodiment of the present application is shown in the structural schematic diagram. As shown in (a) in FIG. 1, Figure 7 As shown in (a) in FIG. 1, the first unit further includes a morphological dilation layer. The morphological dilation layer performs dilation processing on the edge vector to obtain a dilated edge vector, and transmits the dilated edge vector to the residual fusion module. If the first gating signal is output from the adaptive gating layer to the residual fusion module, the residual fusion module performs residual fusion processing on the dilated edge vector and the first vector to obtain a processed vector. If the second gating signal is output from the adaptive gating layer to the residual fusion module, the residual fusion module does not perform residual fusion processing on the dilated edge vector and the first vector, and the processed vector output at this time is the first vector.

[0097] In the embodiments of the present application, by adding a morphological dilation layer in the image recognition model, the recognition integrity and robustness of the model for slender or small objects (such as electric wires) can be improved, so that the initially recognized object region can be effectively expanded and connected, and the missing pixels caused by insufficient model recognition ability or unobvious object features can be compensated. Moreover, the "first recognition and then dilation" mode ensures that the vector transmitted to the subsequent residual fusion module is more complete and continuous, and thus the processed vector output by the residual fusion module can accurately represent the boundary of the object in the RGB image, thereby effectively avoiding the occurrence of object residues in the virtual three-dimensional scene obtained by three-dimensional reconstruction based on the recognition result output by the image recognition model, and improving the visual quality of the virtual three-dimensional scene.

[0098] The accuracy of the analysis of the target object in the RGB image highly depends on the accurate recognition of the region of the target object. If a misjudgment occurs in the recognition of the RGB image, and the pixels of the background or other objects are mistakenly recognized as the pixels of the target object, the subsequent analysis object will be distorted, and the reliability of the entire analysis result will be reduced. For example, for an image including an electric wire, if the shape, position, etc. of the electric wire in the image need to be analyzed, the pixels belonging to the electric wire need to be recognized from the image first, and then the shape, position, etc. of the electric wire can be analyzed based on the pixels belonging to the electric wire. If the pixels belonging to other objects are mistakenly determined as the pixels of the electric wire, the reliability of the analysis result will be reduced.

[0099] Therefore, in order to avoid the above situation, a morphological erosion layer can be arranged in the image recognition model. As shown in (b) of FIG. 1, Figure 7 In some embodiments, the first unit further includes a morphological erosion layer. The morphological erosion is just the opposite of the morphological dilation, and is mainly used to shrink or "thinning" the target region in the image. This operation also uses a "structuring element" as a probe to reduce the target region in the image. For example, in an object recognition task, once the model recognizes that a certain region belongs to a specific object, the morphological erosion layer will remove some edge pixels, thereby shrinking the boundary of the initially recognized region and removing the misjudgment pixels caused by noise or model uncertainty. For example, if the model recognizes that the 5th to 15th pixels in the first row of the RGB image belong to the electric wire, the morphological erosion layer will determine that the 5th to 6th pixels and the 14th to 15th pixels in the row are background, thereby shrinking the electric wire region from the original 5th to 15th pixels to the 7th to 13th pixels.

[0100] Figure 7In the (b) in the morphological erosion layer, the morphological erosion layer performs erosion processing on the edge vector to obtain an eroded edge vector, and transmits the eroded edge vector to the residual fusion module. If the first gating signal is output from the adaptive gating layer to the residual fusion module, the residual fusion module performs residual fusion processing on the eroded edge vector and the first vector to obtain a processed vector. If the second gating signal is output from the adaptive gating layer to the residual fusion module, the residual fusion module does not perform residual fusion processing on the eroded edge vector and the first vector, and the output processed vector is the first vector.

[0101] In the embodiments of the present application, by adding a morphological erosion layer in the image recognition model, the edge of the object preliminarily recognized can be optimized by the morphological erosion layer, and the pixel points misjudged due to noise, texture interference or model overfitting can be actively removed, so that a more accurate edge vector is obtained. Through the above-mentioned manner, the problem of inaccurate boundary caused by excessive expansion of the recognition area can be effectively solved, so as to improve the accuracy of the image recognition model.

[0102] In order to improve the accuracy of the determined loss value and further improve the accuracy of the trained image recognition model, in some embodiments, step 190 includes steps b1-b5.

[0103] Step b1: determining a basic loss value by the parameter optimization module according to the recognition result and the semantic information.

[0104] Among them, the cross-entropy loss value (L CE) and the Dice loss value (L Dice) can be determined first. The sum of the cross-entropy loss value and the Dice loss value is the basic loss value.

[0105] Step b2: determining a connectivity loss value according to the first vector.

[0106] The connectivity loss is a kind of regularization loss, and its purpose is to punish the discontinuous object region in the prediction result. Its core hypothesis is that a real object should form a connected component in the image. Therefore, this loss function quantifies the degree of separation of isolated pixels or pixel clusters from the main body region in the predicted mask. When the model predicts isolated and disconnected "enclaves", a larger loss value will be generated, so as to encourage the model to generate more complete and continuous object regions.

[0107] Specifically, if two kinds of objects are identified by using the image recognition model, the first vector includes the prediction score of each pixel point belonging to various objects, then the connectivity loss value can be determined for each object respectively, and after obtaining two connectivity loss values, the final connectivity loss value can be obtained by weighted sum of the two connectivity loss values.

[0108] Step b3: determining an edge consistency loss value based on the processed vector.

[0109] Specifically, the RGB image is processed by a three-layer convolutional layer to obtain an edge prediction vector, and the edge consistency loss value L_conn is determined according to the edge prediction vector and the processed vector output by the residual fusion module. The edge consistency loss value is used to represent the continuity and coherence of the line formed by the pixel points belonging to the same object. By determining the edge consistency loss value, it can be ensured that the edge finally predicted by the model is consistent with the bottom edge extracted from the original image in spatial position, so as to punish the deviation, break or blur of the predicted edge, and enhance the positioning accuracy and continuity of the object boundary.

[0110] Step b4: determining a total loss value according to the basic loss value, the connectivity loss value and the edge consistency loss value.

[0111] Wherein, the total loss value L_total can be determined by the formula L_total=L_CE+L_Dice+0.3·L_wire+0.2·L_edge+λ_conn·L_conn. L_wire is the weighted cross-entropy for wire pixel points, β=5; L_edge is the Dice loss value calculated on the edge mask M; λ_conn=0.1; and the class weight adopts a medium frequency balance strategy.

[0112] Step b5: adjusting the parameters of the image recognition model according to the total loss value.

[0113] Wherein, the total loss value can be used to adjust the weight, bias and other parameters of the model to make the model converge.

[0114] In the embodiments of the present application, by constructing a multi-dimensional composite loss value composed of a basic loss value, a connectivity loss value and an edge consistency loss value, the fine constraint of the model training process is realized. This method not only uses cross-entropy and Dice loss to ensure the basic accuracy of pixel-level classification, but also introduces connectivity loss to punish the break and isolation of the model in identifying object regions, so as to force the model to learn the overall structure of the object and generate a more complete and continuous predicted region. At the same time, the edge consistency loss value aligns the edge predicted by the model with the bottom edge clue extracted from the original image, effectively suppressing the deviation and blur of the predicted boundary, and significantly improving the positioning accuracy of the object contour. This loss value determination method combining global classification accuracy, topological structure integrity and boundary positioning accuracy can improve the accuracy of the determined total loss value, so that the model can learn more robust and more physical reality spatial features, and finally improve the comprehensive performance and reliability of the trained image recognition model in complex scenes.

[0115] In some embodiments, after step 110, further comprising: adding a line fitting the shape of the certain object in the plurality of objects on the RGB image, and labeling semantic information for pixels corresponding to the line to update the RGB image. Step 140 comprises: inputting the updated RGB image into the first feature extraction unit in the feature extraction module.

[0116] For cases where fine objects need to be recognized, such as fine wires in an image, to improve the ability of the image recognition model to recognize fine objects, a line fitting the shape of the wire can be added to the RGB image, and the wire corresponding line in various backgrounds, thicknesses, gray scales, and non-connected situations can be covered, thereby increasing the learning samples of the wire to improve the training effect of the image recognition model. Moreover, by adding a line fitting the shape of the wire in the RGB image, the operation is simple and the cost is extremely low, without the need for expensive and time-consuming real data collection and labeling. This data enhancement strategy can expand the training set in a controllable manner, and can simulate various extreme or rare situations that may occur in the real world, thereby effectively improving the generalization ability and robustness of the model for fine and linear targets, and ensuring the recognition accuracy and stability of the model in diversified scenarios.

[0117] Figure 8 A flowchart of an image recognition method provided by an embodiment of the present application is shown, which is executed by an electronic device. As shown in Figure 8 The method comprises steps 210-240.

[0118] Step 210: obtaining an image to be recognized.

[0119] The image to be recognized is an image taken of a real three-dimensional scene. The real three-dimensional scene can be taken by an electronic device including a camera, and after obtaining the image to be recognized, the image to be recognized is transmitted to the electronic device executing the embodiment of the present application. Alternatively, if the electronic device executing the embodiment of the present application includes a camera, the real three-dimensional scene is taken by the camera, and after obtaining the image to be recognized, the image to be recognized is transmitted to the processor so that the processor executes the subsequent steps.

[0120] Step 220: dividing the image to be recognized into a plurality of sub-images to be recognized.

[0121] The adjacent sub-images to be recognized have an overlapping area. Figure 9 A schematic diagram of the image to be recognized and the plurality of sub-images to be recognized provided by an embodiment of the present application is shown. As shown in Figure 9As shown, the to-be-recognized image A1B1C1D1 is divided into four to-be-recognized sub-images, which are a first to-be-recognized sub-image A1A2A3A4, a second to-be-recognized sub-image B1B2B3B4, a third to-be-recognized sub-image C1C2C3C4, and a fourth to-be-recognized sub-image D1D2D3D4.

[0122] The overlapping area between the first to-be-recognized sub-image A1A2A3A4 and the second to-be-recognized sub-image B1B2B3B4 is an area A2B2B3A4, the overlapping area between the first to-be-recognized sub-image A1A2A3A4 and the third to-be-recognized sub-image C1C2C3C4 is an area A3C2C3A4, the overlapping area between the third to-be-recognized sub-image C1C2C3C4 and the fourth to-be-recognized sub-image D1D2D3D4 is an area C3D3D2C4, and the overlapping area between the second to-be-recognized sub-image B1B2B3B4 and the fourth to-be-recognized sub-image D1D2D3D4 is an area B3D3D4B4.

[0123] It is worth noting that, Figure 9 The number of to-be-recognized sub-images is not limited.

[0124] Step 230: input the plurality of to-be-recognized sub-images into the trained image recognition model respectively to obtain an identification result corresponding to each to-be-recognized sub-image output by the trained image recognition model.

[0125] In this step, the trained image recognition model refers to a model trained by the foregoing image recognition model training method embodiment.

[0126] Step 240: determine an identification result of the to-be-recognized image according to the identification result corresponding to each to-be-recognized sub-image.

[0127] Since there is an overlapping area between adjacent to-be-recognized sub-images, there may be multiple image recognition results for the overlapping area. When determining the image recognition result of the overlapping area, one of the multiple image recognition results of the overlapping area can be directly determined as the identification result of the same area in the to-be-recognized image. For a non-overlapping area, since there is only one identification result, the identification result is directly taken as the identification result of the same area in the to-be-recognized image. Through the above manner, the identification results of all areas of the to-be-recognized image can be obtained.

[0128] In the embodiment of the present application, by dividing the to-be-recognized image into a plurality of to-be-recognized sub-images and identifying each to-be-recognized sub-image respectively, the problem of reduced recognition accuracy caused by a too large to-be-recognized image or a target object located at the edge of the to-be-recognized image can be effectively solved, thereby improving the accuracy of the image recognition result.

[0129] In Figure 8 On the basis of the provided embodiments, in some embodiments, in step 210, for each overlapping region in each to-be-identified sub-image, the corresponding identification result of the overlapping region is determined from the identification results corresponding to all to-be-identified sub-images containing the overlapping region, a plurality of identification results are obtained, the plurality of identification results are subjected to weighted fusion processing, and a final identification result corresponding to the overlapping region is obtained, which is taken as the identification result of the same region as the overlapping region in the to-be-identified image.

[0130] For example, for the overlapping region A2B2C3D4, since the identification result of the overlapping region is included in the identification results of the first to-be-identified sub-image A1A2A3A4 and the second to-be-identified sub-image B1B2B3B4, the identification results of the overlapping region A2B2C3D4 in the two to-be-identified sub-images are subjected to weighted fusion, and the corresponding final identification result of the overlapping region A2B2C3D4 is obtained.

[0131] For the overlapping region A4B3D3C4, since the identification result of the overlapping region is included in the identification results of the first to-be-identified sub-image A1A2A3A4, the second to-be-identified sub-image B1B2B3B4, the third to-be-identified sub-image C1C2C3C4 and the fourth to-be-identified sub-image D1D2D3D4, the identification results of the overlapping region A4B3D3C4 in the four to-be-identified sub-images are subjected to weighted fusion, and the corresponding final identification result of the overlapping region A4B3D3C4 is obtained. The other overlapping regions are similar to the overlapping region A2B2C3D4 and the overlapping region A4B3D3C4, and will not be introduced one by one.

[0132] In the embodiments of the present application, the plurality of identification results of the overlapping region are subjected to weighted fusion, which can effectively integrate the identification results from different to-be-identified sub-images, and by weighting, the identification result with higher confidence can be given a greater weight, so as to mutually confirm and correct the potential identification deviation, thereby improving the accuracy and reliability of the final identification result of the overlapping region.

[0133] Figure 10 A structure schematic diagram of an electronic device provided by an embodiment of the present application is shown, and the specific implementation of the electronic device is not limited in the specific embodiments of the present application.

[0134] As Figure 10 shown, the electronic device 300 can include a processor 302 and a memory 304.

[0135] The memory 304 is configured to store a computer program 306. The memory 304 can include a high-speed RAM memory, and can further include a non-volatile memory such as at least one disk memory. The computer program 306 can include computer-executable instructions.

[0136] The processor 302 is configured to execute the computer program 306 to implement the above-described image recognition model training method and / or image recognition method embodiments.

[0137] The processor 302 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the electronic device 300 can be the same type of processors, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0138] The embodiments of the present application provide a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-described image recognition model training method and / or image recognition method embodiments.

[0139] The embodiments of the present application provide a computer program, which can be executed by a processor to implement the above-described image recognition model training method and / or image recognition method embodiments.

[0140] The embodiments of the present application provide a computer program product, which includes a computer program. The computer program is executed by a processor to implement the above-described image recognition model training method and / or image recognition method embodiments.

[0141] In several embodiments provided in the present application, any function, if implemented in the form of a software function module / unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or an electronic device) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media capable of storing computer program codes.

[0142] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present application as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0143] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a claim enumerating several means, several of those means can be embodied by one and the same item of hardware. The terms 'first','second', 'third', etc. do not denote any order. They are used to distinguish between different sets of elements. The steps of the methods described herein do not have to be performed in the order described, unless otherwise specified.

[0144] The embodiments described above are merely illustrative for the present application and they do not limit the scope of the present application. Any modifications and improvements made to the present application by those skilled in the art are to be within the scope of the present application. Therefore, the scope of the present application is to be determined by the appended claims rather than the foregoing description.

Claims

1. A method for training an image recognition model, characterized in that, The method is used for training an image recognition model comprising a feature extraction module, a feature fusion module, a decoder, an identification result generation module and a parameter optimization module, the image recognition model being used for identifying a plurality of objects, the method comprising: obtaining an RGB image, wherein the RGB image is an image obtained by photographing a target region in a real three-dimensional scene, and each pixel point in the RGB image is labeled with semantic information; obtaining point cloud data collected from the target region in the real three-dimensional scene; projecting the point cloud data into an image coordinate system to obtain a projection image; inputting the RGB image into a first feature extraction unit in the feature extraction module to perform feature extraction on the RGB image, thereby obtaining a first feature map set, wherein the first feature map set comprises first feature maps of a plurality of scales; inputting the projection image into a second feature extraction unit in the feature extraction module to perform feature extraction on the projection image, thereby obtaining a second feature map set, wherein the second feature map set comprises second feature maps of each scale in the plurality of scales; performing feature fusion on first feature maps and second feature maps of the same scale in the first feature map set and the second feature map set by the feature fusion module, thereby obtaining a plurality of fusion feature maps corresponding to the plurality of scales; performing identification and prediction on the RGB image according to the plurality of fusion feature maps by the decoder, thereby obtaining a first vector, wherein the first vector comprises prediction scores of each pixel point in the RGB image belonging to various objects in the plurality of objects; generating an identification result based on the first vector by the identification result generation module, wherein the identification result comprises an object category to which each pixel point in the RGB image belongs; adjusting parameters of the image recognition model based on the identification result and the semantic information by the parameter optimization module.

2. The method of claim 1, wherein, The plurality of fusion feature maps comprises a first scale fusion feature map, a second scale fusion feature map, a third scale fusion feature map and a fourth scale fusion feature map; The method comprises: performing up-sampling processing on the first scale fusion feature map to obtain a second scale third feature map; performing fusion processing on the second scale fusion feature map and the second scale third feature map to obtain a second scale fourth feature map; performing up-sampling processing on the second scale fourth feature map to obtain a third scale fifth feature map; performing fusion processing on the third scale fusion feature map and the third scale fifth feature map to obtain a third scale sixth feature map; performing up-sampling processing on the third scale sixth feature map to obtain a fourth scale seventh feature map; performing fusion processing on the fourth scale fusion feature map and the fourth scale seventh feature map to obtain a fourth scale eighth feature map; performing identification and prediction on the RGB image according to the fourth scale eighth feature map to obtain the first vector.

3. The method of claim 1, wherein, The image recognition model further comprises an edge feature optimization module, the edge feature optimization module comprising a first unit, a second unit and a residual fusion module, the first unit comprising an activation function layer, three convolutional layers, an edge feature determination layer, the second unit comprising a confidence determination layer and an adaptive gating layer; Before the identification result is generated based on the first vector by the identification result generation module, the method further comprises: processing the first vector by the activation function layer in the first unit to obtain a second vector; extracting edge features from the second vector by the three convolutional layers to obtain a third vector; determining edge features from the third vector by the edge feature determination layer to obtain an edge vector; determining the confidence of the first vector by the confidence determination layer; outputting a gating signal to the residual fusion module by the adaptive gating layer according to the confidence of the first vector; performing residual fusion processing on the edge vector and the first vector by the residual fusion module according to the gating signal to obtain a processed vector; the identification result is generated based on the first vector by the identification result generation module, comprising: generating the identification result by the identification result generation module based on the processed vector.

4. The method of claim 3, wherein, The first unit further comprises a morphological dilation layer; before the residual fusion processing is performed on the edge vector and the first vector by the residual fusion module according to the gating signal to obtain a processed vector, the method further comprises: performing dilation processing on the edge vector by the morphological dilation layer to obtain a dilated edge vector; the residual fusion processing is performed on the edge vector and the first vector by the residual fusion module according to the gating signal to obtain a processed vector, comprising: performing residual fusion processing on the dilated edge vector and the first vector by the residual fusion module according to the gating signal to obtain the processed vector; or The first unit further comprises a morphological erosion layer, and before the residual fusion processing is performed on the edge vector and the first vector by the residual fusion module according to the gating signal to obtain a processed vector, the method further comprises: performing erosion processing on the edge vector by the morphological erosion layer to obtain an eroded edge vector; the residual fusion processing is performed on the edge vector and the first vector by the residual fusion module according to the gating signal to obtain a processed vector, comprising: performing residual fusion processing on the eroded edge vector and the first vector by the residual fusion module according to the gating signal to obtain the processed vector.

5. The method of claim 3, wherein, the parameters of the image recognition model are adjusted based on the identification result and the semantic information by the parameter optimization module, comprising: determining a basic loss value according to the identification result and the semantic information by the parameter optimization module; determining a connectivity loss value according to the first vector; determining an edge consistency loss value based on the processed vector; determine a total loss value according to the base loss value, the connectivity loss value and the edge consistency loss value; adjust parameters of the image recognition model according to the total loss value.

6. The method of claim 1, wherein, After the RGB image is acquired, the method further includes: adding lines on the RGB image, the lines being adapted to shapes of objects in the plurality of objects, and labeling semantic information for pixel points corresponding to the lines to update the RGB image; the inputting the RGB image into the first feature extraction unit in the feature extraction module includes: the inputting the updated RGB image into the first feature extraction unit in the feature extraction module.

7. An image recognition method characterized by, The method includes: acquiring an image to be recognized; dividing the image to be recognized into a plurality of sub-images to be recognized, wherein adjacent sub-images to be recognized have overlapping areas; inputting the plurality of sub-images to be recognized into a trained image recognition model respectively to obtain recognition results corresponding to each of the sub-images to be recognized output by the trained image recognition model, wherein the trained image recognition model is trained by the image recognition model training method of any one of claims 1-6; determining a recognition result of the image to be recognized according to the recognition results corresponding to each of the sub-images to be recognized.

8. The method of claim 7, wherein, The determining a recognition result of the image to be recognized according to the recognition results corresponding to each of the sub-images to be recognized includes: taking a recognition result corresponding to a non-overlapping area in each of the sub-images to be recognized as a recognition result of an area in the image to be recognized that is the same as the non-overlapping area; for each overlapping area in the sub-images to be recognized, determining a recognition result corresponding to the overlapping area from recognition results corresponding to all sub-images to be recognized containing the overlapping area, obtaining a plurality of recognition results, performing weighted fusion processing on the plurality of recognition results to obtain a final recognition result corresponding to the overlapping area, and taking the final recognition result as a recognition result of an area in the image to be recognized that is the same as the overlapping area.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein, The processor executes the computer program to implement the image recognition model training method of any one of claims 1-6 and / or the image recognition method of any one of claims 7-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the image recognition model training method of any one of claims 1-6 and / or the image recognition method of any one of claims 7-8.

Images