Lightweight semantic understanding method for target key part and related equipment

Abstract

The invention belongs to the field of visual semantic understanding, and discloses a lightweight semantic understanding method for a target key part and related equipment, a lightweight semantic understanding model in the method fuses features from top to bottom by using a feature pyramid network, retains low-level details and high-level semantic information at the same time, and improves the semantic understanding efficiency. Performing primary processing on the target key part image, and inputting an obtained feature map into a target detection head and a semantic segmentation model to realize parallel processing; the target detection head can identify bounding boxes and categories of different targets in the image, and the semantic segmentation model provides category information of a pixel level; when the target detection head and the semantic segmentation model are combined, the target detection head can utilize rich context information of the semantic segmentation model to reduce the probability of false detection, and meanwhile, the semantic segmentation model can utilize positioning information of the target detection head to refine segmentation boundaries, so that a mutual assistance effect is achieved; by adopting the method, the stability of an output result is ensured, and the applicability and robustness of semantic understanding are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, specifically to the field of visual semantic understanding, and in particular to a lightweight semantic understanding method for key parts of a target and related equipment. Background Art

[0002] Visual semantic understanding refers to assigning each pixel of an image to a corresponding category, segmenting several regions with semantic meaning, and realizing semantic understanding and analysis of the image. With the continuous enrichment of aviation equipment and the continuous expansion of its application scenarios, the application value and prospects of image semantic segmentation in military and civil aviation are becoming more and more obvious, especially the semantic understanding of key parts of the target is more urgent. Semantic understanding of key parts of the target can achieve more accurate and efficient target identification and tracking, lock key information, help the automation system to efficiently understand and analyze image data, and improve aircraft safety and efficiency. It can be seen that the semantic understanding of key parts of the target can effectively promote the intelligentization of aviation science and technology, and at the same time has broad and important application prospects.

[0003] Semantic understanding of key parts of an object is a special semantic segmentation algorithm. It focuses on the information of object components. It divides an object into several semantic parts in order to provide more detailed information for specific tasks. For example, when an autonomous robot repairs a car engine, it first needs to further segment the car into tires, hoods, doors, etc. At present, multi-category object-level semantic segmentation has been widely studied, but object component-level semantic segmentation has not been fully explored, and existing work mainly targets some specific categories with precise positioning, such as people and cars. Current methods can be roughly divided into two categories. The first category usually focuses on exploring the intrinsic relationship and structural information of object parts. For example, Liang et al. proposed a self-supervised loss function to maintain the parsing structure. Wang et al. established a hierarchical tree structure to construct components based on the spatial relationship of basic boundary landmarks. Some other studies also use additional structural information, such as human posture and 3D information. The second category focuses on improving the parsing resolution in images or feature maps. For example, Chieh et al. proposed an attention model to fuse the parsing results of different image scaling. Xia et al. proposed a two-level network to fuse global features with detected local features. Although the above models are effective, they are only applicable to single-category situations. With the increasing number of multi-category semantic component annotations, Hariharan et al. proposed a sequential object and component segmentation, where the object mask is first segmented and then partial labels are assigned to the pixels within the mask. However, the predicted semantic object mask error can propagate to the component, and the final component segmentation effect is average. Later, Adobe used a two-way fully convolutional neural network to simultaneously predict the target category and segment the target component under multiple categories. Although the accuracy of component segmentation has improved, the accuracy of the target component category is not high for scenes with similar appearance and texture between categories. Recently, Zhao et al. proposed a joint parsing structure with boundary perception and semantic perception, using the boundary perception module to handle component-level ambiguity and the semantic perception module to solve the category ambiguity problem. Although this method further improves the accuracy of the algorithm, unfortunately, this method still has the problem of component-level semantic ambiguity (recognizing a dog's head as a horse's head), and the algorithm has not been optimized for the characteristics of data collected from multiple angles.

[0004] In addition, traditional semantic segmentation models usually have a large number of parameters and computational complexity, which makes it difficult to deploy the model on mobile or embedded devices. At the same time, the inference speed is slow and cannot meet the real-time requirements. With the rapid development of artificial intelligence technology, more and more mobile and embedded devices have begun to integrate artificial intelligence functions, such as mobile phones, tablets, drones, micro-spacecraft, etc. The computing resources of these devices are limited and cannot support the operation of traditional semantic segmentation models. Therefore, lightweight image semantic segmentation technology has become one of the current research hotspots. In recent years, researchers have proposed a variety of lightweight image semantic segmentation technologies and achieved remarkable results. For example, DeepLabV3+ reduces the number of model parameters and computational complexity by removing redundant parameters or neurons in the model, but this may cause the model's performance on semantic segmentation tasks to decline, especially in the segmentation of details and edges. Models such as XNOR-Net and BinaryConnect use weight quantization technology to convert the model's weights from high-precision floating-point numbers to low-precision integers, thereby reducing the number of model parameters and storage space, but this will reduce the network's representation ability and may cause the model's performance on complex tasks to decline. MobileNet, ShuffleNet, etc. use network structure search technology to automatically search for lighter network structures, but this may limit the network's ability to learn complex features and lack model generalization capabilities. The above semantic segmentation methods have greatly promoted the research progress of lightweight semantic understanding models, but there are still some shortcomings when dealing with complex scenarios. Therefore, for different data sets and application scenarios, targeted customized designs are needed to meet specific task requirements.

[0005] In a multi-view environment, the collected data will have changes in shape, rotation, illumination, and scale, which will put higher requirements on the semantic segmentation of multi-category target components, resulting in the inability of existing methods to solve the semantic understanding problem of key parts of targets. The main difficulties are: First, the inter-class semantic ambiguity caused by multiple key parts. Early research mainly focused on the segmentation of key parts of specific single-category targets (such as human bodies). Although research in recent years has begun to involve the segmentation of multiple key parts, due to the similarity of contours, textures, or colors of the same parts of different categories, category ambiguity has become a more difficult problem, especially in scenes with similar targets and occlusions, which is more obvious and is also an important factor leading to poor robustness of semantic understanding methods. Second, the problem of image feature attribute changes caused by multi-view data; such as scale, rotation, illumination changes and occlusion, or even a combination of multiple feature transformations, which causes image feature attributes to change, posing challenges to the applicability and robustness of existing semantic segmentation algorithms trained on conventional view data sets. It can be seen that the existing semantic understanding technology of target key parts has poor output stability due to the inter-class semantic ambiguity caused by the existence of multiple types of key parts, which affects the applicability and robustness of semantic understanding. Summary of the invention

[0006] The present invention provides a lightweight semantic understanding method and related equipment for target key parts, so as to solve the technical problem that the existing semantic understanding technology for target key parts has inter-class semantic ambiguity caused by the existence of multiple types of key parts, resulting in poor stability of output results, affecting the practicality and robustness of semantic understanding.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: A lightweight semantic understanding method for key parts of a target, comprising: Collect images of key parts of the target; Input the target key part image into the pre-trained lightweight semantic understanding model, and output the target detection result and semantic segmentation result; Among them, the lightweight semantic understanding model is trained based on images of key parts of historical targets, including a lightweight backbone network that uses a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature maps extracted by the lightweight backbone network in parallel.

[0008] Furthermore, the images of key parts of the historical targets include network images and rendered images; The steps to obtain the rendered image include: Obtain 3D models corresponding to various targets; The 3D model is processed by changing the camera's perspective, position, distance, and background to obtain a rendered image.

[0009] Furthermore, the lightweight backbone network performs feature extraction processing on the target key part image and outputs a multi-scale feature map; wherein the scale sizes of the feature map are 1 / 8, 1 / 16 and 1 / 32 of the target key part image respectively.

[0010] Further, the object detection head includes an anchor frame layer, a detection layer and a prediction layer; The anchor frame layer is used to adapt to objects of different sizes in the feature map according to preset anchor frames of different proportions and sizes, so as to generate candidate frames for objects of different sizes on the feature map; The detection layer includes a convolution layer for processing feature maps and extracting features, and can adapt to different detection tasks by adjusting the number of channels and the kernel size; The prediction layer includes a convolutional layer and an output layer, which is used to predict the bounding boxes and categories of different targets based on the extracted features; wherein the number of bounding boxes and the number of categories predicted are determined by the number of channels of the output layer.

[0011] Furthermore, a non-maximum suppression algorithm is used to suppress the output bounding box to retain the most representative bounding box.

[0012] Furthermore, the semantic segmentation model adopts a PSPNet network model, and PSPNet introduces a pyramid pooling module for fusing context information of different scales.

[0013] Furthermore, in the loss function design process of the lightweight semantic understanding model, the weight of the target detection loss and the weight of the semantic segmentation loss are adjusted to control the proportion of the two in training. The specific formula is as follows:

[0014] in, represents the total loss of the lightweight semantic understanding model; represents semantic segmentation loss; represents the target detection loss; Represents the semantic segmentation loss ratio; Represents the target detection loss ratio.

[0015] A lightweight semantic understanding system for target key parts, used to implement the steps of the above-mentioned lightweight semantic understanding method for target key parts, including: An image acquisition module, used to acquire images of key parts of the target; The semantic understanding module is used to input the target key part image into the pre-trained lightweight semantic understanding model and output the target detection result and semantic segmentation result; Among them, the lightweight semantic understanding model is trained based on images of key parts of historical targets, including a lightweight backbone network that uses a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature maps extracted by the lightweight backbone network in parallel.

[0016] A device comprising: Memory for storing computer programs; A processor is used to implement the steps of the above-mentioned lightweight semantic understanding method for key parts of the target when executing the computer program.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the steps of the above-mentioned lightweight semantic understanding method for key target parts.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a lightweight semantic understanding method for target key parts. The method uses a lightweight semantic understanding model to process the target key part image to obtain the target detection result and the semantic segmentation result; the lightweight semantic understanding model uses a feature pyramid network to fuse features from top to bottom, while retaining low-level details and high-level semantic information, preliminarily processes the target key part image and inputs the obtained feature map into a target detection head and a semantic segmentation model to achieve parallel processing; since the target detection head can identify the bounding boxes and categories of different targets in the image, and the semantic segmentation model provides pixel-level category information, it can distinguish in detail the category to which each pixel in the image belongs. Therefore, in the process of combining the two, the target detection head can use the rich context information of the semantic segmentation model to reduce the probability of false detection, and the semantic segmentation model can use the positioning information of the target detection to refine the segmentation boundary, which plays a mutually auxiliary effect; the method effectively solves the semantic ambiguity problem between classes caused by multiple types of key parts, ensures the stability of the output results, and improves the applicability and robustness of semantic understanding.

[0019] Preferably, in the present invention, in order to solve the problem of image feature attribute changes caused by multi-view data, during the data collection process, the method obtains network pictures as relevant real data through network collection methods on the one hand; on the other hand, the 3D model is rendered and virtual data is supplemented by using 3dsmax software to change the camera perspective, position, distance and change the model background, so that the data generation process can be fully controlled. Specifically, it includes various factors such as the shape, size, color, texture, lighting, angle and so on of the object. Such control can help generate very accurate and consistent data for training and testing deep learning models; it solves the problems of uneven distribution of multi-view scene data, high acquisition cost and poor diversity from the data level, and further improves the applicability and robustness of the method.

[0020] Preferably, in the present invention, a lightweight backbone network is used to perform feature extraction processing on the image of key parts of the target to output multi-scale feature maps, thereby improving the model's adaptability to targets of different scales; by outputting multi-scale feature maps, the model can simultaneously capture the key details and overall structural information of the target, thereby more accurately identifying targets of different sizes. This multi-scale feature fusion method enhances the flexibility of the model, making it more efficient and accurate when processing complex scenes.

[0021] Preferably, in the present invention, the target detection head enables the model to dynamically adjust the generation and feature extraction strategies of candidate boxes according to targets of different sizes through the coordinated work of the anchor box layer, the detection layer and the prediction layer, thereby improving the detection accuracy; Further preferably, the application of the non-maximum suppression algorithm effectively suppresses redundant bounding boxes, retains the most representative detection results, and further improves the practicality of the model.

[0022] Preferably, in the present invention, the semantic segmentation model adopts the PSPNet network model. PSPNet introduces a pyramid pooling module, which effectively integrates context information of different scales, can more accurately identify different objects and semantic information in the scene, and significantly improve the accuracy of scene parsing. In addition, PSPNet also uses depthwise separable convolution, which decomposes the standard convolution into depthwise convolution and pointwise convolution, thereby significantly reducing the model complexity, effectively reducing the model parameter amount and calculation amount, and improving the training and reasoning efficiency of the model.

[0023] Preferably, in the present invention, this method achieves a balanced optimization between the target detection loss and the semantic segmentation loss by adjusting the weights of the loss function, thereby improving the overall performance of the model; this weight adjustment mechanism enables the model to pay more attention to the learning of important tasks during the training process, thereby ensuring detection accuracy while also obtaining better segmentation effects; the training strategy of this method helps the model to better adapt to different application scenarios and needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The embodiment of the present invention provides a display of some pictures collected on the Internet; Figure 2 A comparison diagram of a network image and a rendered image provided by an embodiment of the present invention; wherein the left image is a network image; and the right image is a rendered image; Figure 3 An example of a picture generated by adjusting the viewing angle provided by an embodiment of the present invention; Figure 4 An example of a picture generated by scaling and rotating a model provided in an embodiment of the present invention; wherein the upper picture is a scaled picture; and the lower picture is a rotated picture; Figure 5 An example of an airplane picture rendered by adjusting a model background provided by an embodiment of the present invention; Figure 6 An example of target detection and semantic segmentation annotation provided by an embodiment of the present invention, the left picture is target detection; the right picture is semantic segmentation; Figure 7 A schematic diagram of a semantic understanding model architecture of a target key part under multiple perspectives provided by an embodiment of the present invention; Figure 8 A channel information diagram in a target detection head provided by an embodiment of the present invention; Fig. 9 The semantic understanding model architecture diagram provided by the embodiment of the present invention; Fig.10 The inference results of the semantic understanding model of aircraft-type targets provided by the embodiment of the present invention are displayed; Fig.11 The reasoning results of the semantic understanding model of automobile objects provided by the embodiment of the present invention are displayed; Fig.12 The reasoning results of the semantic understanding model of ship-type targets provided by the embodiment of the present invention are displayed; Fig.13 The tank target semantic understanding model reasoning results provided by the embodiment of the present invention are displayed; Fig.14 The satellite target semantic understanding model reasoning results provided by the embodiment of the present invention are displayed; Fig.15 The average miou values ​​of ten key parts of five types of targets in the test set provided by the embodiment of the present invention; Fig.16 Examples of maximum and minimum scale images of cars and ships provided in the embodiments of the present invention; Fig.17 The miou value of aircraft targets of different scales provided by the embodiment of the present invention varies with scale; Fig.18 A graph showing the change in miou values ​​of automobile targets of different scales as a function of scale provided by an embodiment of the present invention; Fig.19 A graph showing the change in miou values ​​of ship-like targets of different scales as a function of scale provided by an embodiment of the present invention; Fig. 20 A graph showing the change in miou value of tank targets of different scales as a function of scale provided by an embodiment of the present invention; Fig.21 The miou value of satellite targets of different scales provided by the embodiment of the present invention varies with scale; Fig. 22 A flowchart of a lightweight semantic understanding method for key target parts provided by the present invention; Fig.23 A schematic diagram of the structure of a lightweight semantic understanding system for target key parts provided by the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the embodiments and drawings: Example 1 Combined with the background technology, the current development background of semantic understanding technology is further introduced below: Before deep learning was widely used, researchers used random forests and conditional random fields to build classifiers for semantic learning. In recent years, deep learning-based methods have become the mainstream solution for semantic segmentation. According to the characteristics of the methods and the processing granularity, they can be divided into two categories: region-based and pixel-based image semantic segmentation techniques. Among them, although the region-based method has achieved certain segmentation effects, it is not ideal in terms of segmentation accuracy and speed. Therefore, researchers proposed a pixel-based semantic segmentation method. It can be divided into two categories based on the annotation type and learning method: fully supervised learning and weakly supervised learning methods. At present, most semantic segmentation methods based on deep learning belong to the fully supervised learning type. In the training stage, convolutional networks are used to extract semantic information from pixel-level annotated images and then classify them. In order to solve the problem of semantic-level image segmentation, Shelhamer et al. designed a classic fully convolutional network (FCN). The network classifies images at the pixel level in a fully supervised learning manner and uses a deconvolution layer to model the last layer of the rotation. It not only retains the spatial information of the original input image, but also generates a prediction for each pixel, and finally realizes pixel classification. Of course, FCN has two key problems that need to be solved in this field: after pooling, the resolution of the feature map will continue to decrease, resulting in the loss of spatial position information of some pixels; due to insufficient consideration of image context information, the spatial position information cannot be fully utilized, resulting in the inability to balance the use of local features and global features. So far, researchers have proposed a series of new methods to solve problems such as discontinuous FCN segmentation boundaries and rough results. From the perspective of the improved characteristics of the algorithm, these methods can be divided into 7 categories: FCN-based methods, methods based on optimized convolutional structures, encoder-decoder-based methods, probabilistic graph model-based methods, feature fusion-based methods, RNNs-based methods, and GAN-based methods.

[0026] The fully supervised learning method has brought about a leap forward in the field of AI, but this progress relies on a large amount of fully supervised annotated data. Since people often lack sufficient prior knowledge in real life, it costs a lot to obtain pixel-level annotations. Weakly supervised semantic segmentation came into being. This method is a way to use relatively easier-to-obtain annotations as supervision to train semantic segmentation models. According to the different types of weak annotations used, the research on weakly supervised semantic segmentation can be roughly divided into four directions: bounding box-level annotation, that is, marking the position and label of the target object with a rectangular box; point and line level annotation, that is, marking the position and label of the target object with a point or a line; image-level annotation, that is, making a picture correspond to a label; hybrid annotation method, that is, using a combination of the above three methods.

[0027] At present, the existing semantic understanding algorithms have not been optimized for the characteristics of data collected from multiple angles. In a multi-view environment, the collected data will have changes in shape, rotation, illumination, and scale, which will put higher requirements on the semantic segmentation of multiple types of target components, resulting in the inability of existing methods to solve the problem of semantic understanding of key parts of the target. The main difficulties are: First, the problem of image feature attribute changes caused by multi-view data. For example, scale, rotation, illumination changes and occlusion, or even a combination of multiple feature transformations, cause image feature attributes to change, which challenges the applicability and robustness of existing semantic segmentation algorithms trained based on conventional view data sets. Second, the problem of inter-class semantic ambiguity caused by multiple key parts. Early studies mainly focused on the segmentation of key parts of specific single-type targets (such as human bodies). Although research in recent years has begun to involve the segmentation of multiple key parts, due to the similarity of contours, textures or colors of the same parts of different types, category ambiguity has become a more difficult problem, especially in scenes with similar targets and occlusions. It is more obvious, and it is also an important factor leading to the poor robustness of semantic understanding methods. It can be seen that the existing methods still have certain limitations and need further in-depth analysis and research. Therefore, it is very necessary to propose a method that can semantically understand the key parts of different types of targets under multi-view conditions.

[0028] In order to solve the above problems, this embodiment provides a lightweight semantic understanding method for key parts of a target, which specifically includes the following steps: 1. The training of lightweight semantic understanding model includes the following steps: Step 1: Data collection: In this embodiment, a total of 10,000 original images were collected, covering 5 types of sea, land, air and space targets, and 10 key parts, which have certain military reference significance, as shown in Table 1; Table 1 is the data collection table

[0029] It can be seen that this embodiment collects the above-mentioned network pictures as images of key parts of historical targets; this method forms an initial data set by collecting a large number of network pictures. This part of the data covers various themes, scenes and objects. This diversity helps to train a model that is more robust to various changes in the real world. However, pictures collected on the Internet often have problems such as a single perspective, uneven picture size and resolution. Therefore, this method uses 3dsmax software to render a 3D model to generate another part of the data as a supplement to the existing data set. Using 3D modeling and rendering tools, the data generation process can be fully controlled. Specifically, Figure 2 , Figure 3 , Figure 4 and Figure 5As shown; including multiple factors such as the shape, size, color, texture, lighting, angle, etc. of the object, this controllability can help generate very accurate and consistent data for training and testing deep learning models.

[0030] Step 2: Data blending.

[0031] In this embodiment, this step determines the ratio of network images and rendered images to be used; this depends on the quantity and quality of each type of data and the degree to which the model relies on these data. This embodiment initially used a 1:1 data ratio, and after multiple tests and adjustments, the ratio was modified to 2:3 (network images: rendered images).

[0032] Step 3: Data labeling.

[0033] In this embodiment, Figure 6 As shown, the samples were manually labeled using the labelme tool. Each sample contains two types of annotations, namely object-level target detection and key part semantic understanding, with a total of 20,000 annotation files.

[0034] At this point, all images of key parts of historical targets have been obtained, and a data set for training a lightweight semantic understanding model has been obtained based on a large number of images of key parts of historical targets through image preprocessing.

[0035] The second step is to design the initial structure of the lightweight semantic understanding model, which includes: Step 1: Overall network architecture: like Figure 7 As shown, the model used in this embodiment adopts a lighter backbone network, and uses the network layer outputs of 1 / 8, 1 / 16, and 1 / 32 of the input image as the input of the Feature Pyramid Net (FPN). FPN can simultaneously process multi-scale low-level, medium-level, and high-level features. The image processed by FPN is used as the input of the head network. The head network in this embodiment uses a method in which target detection and semantic understanding assist each other, processes semantic understanding and target detection tasks in parallel, and generates the results of target detection and semantic understanding of key parts at the same time. Step 2, design of target detection head: The target detection head is mainly responsible for performing multi-scale target detection on the feature map extracted by the backbone network. The target detection head in this embodiment mainly includes three parts: Anchors: Anchor boxes are a set of predefined bounding boxes used to generate candidate boxes on feature maps. This embodiment adapts to objects of different sizes by defining anchor boxes of different scales and sizes in advance.

[0036] Convolutional Layers: The detection layer contains a series of convolutional layers for processing feature maps and extracting features. These convolutional layers can adapt to different detection tasks by adjusting the number of channels and kernel size.

[0037] Prediction Layers: Each prediction layer is responsible for predicting a set of bounding boxes and categories. Each prediction layer usually consists of a convolutional layer and an output layer. The number of channels and shape of the output layer determines the number of bounding boxes and categories predicted.

[0038] Non-Maximum Suppression (NMS): In the output bounding box, the non-maximum suppression algorithm is used to suppress overlapping bounding boxes and only retain the most representative bounding boxes.

[0039] like Figure 8 As shown in the figure, the specific channel information in the target detection head is output as three feature maps of 30×h×w, 30×2h×2w and 30×4h×4w. The smallest size 30×h×w is responsible for detecting large targets, 30×2h×2w is responsible for detecting medium targets, and the largest size 30×4h×4w is responsible for detecting small targets. The number of channels is 30=3×(5+80), where 3 represents 3 anchors, one 5 represents four position information (x, y, w, h) and a confidence (indicating the probability that an object may exist in the current grid), and the other 5 represents the probability of the five categories in this dataset.

[0040] Step 3, design of semantic segmentation model: like Fig. 9 As shown, the semantic understanding model of this embodiment adopts the PSPNet network model. PSPNet introduces a pyramid pooling module, divides the feature map into multiple regions, and performs a pooling operation on each region to generate context feature maps of different scales. These feature maps are then spliced ​​together as the final feature representation, thereby effectively integrating context information of different scales, and can more accurately identify different objects and semantic information in the scene, significantly improving the accuracy of scene parsing. In addition, PSPNet also uses deep separable convolution, which decomposes the standard convolution into deep convolution and point-by-point convolution, thereby significantly reducing the complexity of the model, effectively reducing the number of model parameters and the amount of calculation, and improving the training and reasoning efficiency of the model.

[0041] Step 4, head network connection: In this embodiment, a feature pyramid network (FPN) is used to fuse features from top to bottom, while retaining low-level details and high-level semantic information, and outputting feature maps of three sizes, which are 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively. These feature maps are simultaneously input into the target detection head and the semantic segmentation model in a parallel manner for joint training, and iterative optimization of parameters begins. Since the target detection head can identify the boundaries and categories of different targets in the image, but does not describe the internal structure of the target in detail, and the semantic segmentation model provides pixel-level category information, it can distinguish in detail the category to which each pixel in the image belongs. Therefore, when the two are combined, target detection can use the rich contextual information from the semantic segmentation model to reduce the probability of false detection, while the semantic segmentation model can use the positioning information of target detection to refine the segmentation boundaries, which has a mutually complementary effect.

[0042] Step 5, establishment of loss function: Since the focus of this embodiment is to propose a semantic understanding method for the key parts of the target, when designing the loss function, this embodiment controls the proportion of detection loss and segmentation loss in training by adjusting the weights of the two. After experimental testing, the final detection and segmentation ratio is determined to be 0.4:0.6. At this ratio, the model can better complete the two tasks at the same time and achieve a mutually auxiliary effect.

[0043] The specific formula is as follows:

[0044] in, represents the total loss of the lightweight semantic understanding model; represents semantic segmentation loss; represents the target detection loss; Represents the semantic segmentation loss ratio; Represents the target detection loss ratio.

[0045] At this point, the lightweight semantic understanding model is initialized according to the above structure to prepare for subsequent model training.

[0046] Step 3: Train a lightweight semantic understanding model: Step 1, data set division: In this embodiment, the 10,000 images used for training are divided into three data sets: training set, validation set and test set, with 7,000, 2,500 and 500 images respectively. All types of targets in all data sets have the same proportion, and pictures from different data sources are evenly mixed.

[0047] Step 2, determination of pre-training weights: In this embodiment, the pre-trained weights of the YOLOv5 model based on the COCO dataset are used as a starting point to accelerate the model convergence speed and improve the generalization ability of the model; by using these weights, the training time is significantly reduced, and excellent performance is achieved in the detection and segmentation tasks.

[0048] Step 3: Verification and analysis of the lightweight semantic segmentation model for key target parts: like Fig.10 , Fig.11 , Fig.12 , Fig.13 , Fig.14 As shown, the model's reasoning results for 5 types of targets are shown, among which, Figures 10 to 14 In the figure, (a) is the original image; (b) is the key part semantic segmentation result; (c) is the key part target detection result; (d) is the key part target detection and semantic segmentation result; it can be seen from the reasoning results that this model can clearly segment the key parts of each type of target, and can accurately mark different key parts and the target object as a whole with detection boxes. Fig.13 The MIOU (mean intersection and union) results of the multi-view semantic understanding reasoning results of the images in the test set are displayed. As shown in Figure 13, this embodiment uses the established semantic segmentation model to reason about the 100 images of each type of target in the test set, and calculates the average MIOU value of all images of the same type of target; the results show that the average MIOU values ​​of the five types of targets proposed in the solution in the test set are all greater than 0.85, which has high practical value.

[0049] Among them, the miou value (mean intersection over union) is a commonly used evaluation indicator in semantic segmentation tasks, which is used to measure the degree of consistency between the segmentation results of the model and the true annotation. It is the mean of the iou (intersection over union) of all categories; iou is an indicator of the overlap between the segmentation area predicted by the calculation model and the true segmentation area.

[0050]

[0051] TP (True Positive) indicates the number of pixels correctly predicted by the model as positive, that is, the number of pixels whose true annotation and prediction results are both of this class; FP (False Positive) indicates the number of pixels that the model incorrectly predicts as the positive class, that is, the number of pixels that are actually labeled as other categories but predicted as this class; FN (False Negative) indicates the number of pixels that the model incorrectly predicts as negative classes, that is, the number of pixels that are actually labeled as this class but predicted as other classes.

[0052] Miou is the average value of iou of all categories, which is used to measure the overall performance of the model in multi-category segmentation tasks. The calculation formula is:

[0053] Where N represents the total number of categories; represents the i-th target type .

[0054] Step 4: Perform semantic understanding processing on the target key part images based on the lightweight semantic understanding model: Collect images of key parts of the target; The images of the key parts of the target are input into the pre-trained lightweight semantic understanding model, and the target detection results and semantic segmentation results are output.

[0055] In order to analyze the influence of different target scales on the inference accuracy of the lightweight semantic understanding model established by this method, Fig.14 As shown, in this embodiment, 15 images of the same angle scale of five types of targets from large to small are obtained by rendering with 3dsmax software. The influence of scale change on the model reasoning result is obtained by analyzing the reasoning results of images of different scales. The size of all images is 640×480 pixels. The scale of the target is obtained by calculating the proportion of the Bounding Box obtained by detecting the entire target in the entire image, as shown in Figures 15 to 21 As shown, the impact of scale changes of different categories of targets on the reasoning results can be analyzed.

[0056] By analyzing the results of inference on 15 images of different scales for each type of target, it can be found that for the rendered image, when the scale of the target in the image is ≥0.1 (the ratio of the target detection box to the entire image), this model can stably output the result of miou≥0.85.

[0057] It can be seen that this embodiment provides a lightweight semantic understanding method for key parts of a target, which has the following advantages over existing semantic understanding technologies: First, this method starts with the construction of a balanced data set, obtains relevant real data through network collection methods, and uses the image 3D model rendering method, that is, using 3dmax software to change the camera perspective, position, distance and change the model background to supplement the virtual data, solving the problems of uneven distribution of multi-view scene data, high collection cost and poor diversity from the data level. Second, this method proposes a lightweight semantic understanding model for the key parts of the target, using target detection and segmentation to assist each other, parallel processing of target detection and semantic segmentation tasks, and obtaining the results of target detection and semantic segmentation of key parts at the same time. Compared with traditional methods, it has more complex functions, and when the proportion of the target in the image (pixel 640×480) is ≥0.1, this method can stably output the inference result of miou≥0.85.

[0058] Example 2 The present invention provides a lightweight semantic understanding method for key target parts, such as Fig. 22 As shown, the following steps are included: S1: Collect images of key parts of the target; S2: Input the target key part image into the pre-trained lightweight semantic understanding model, and output the target detection result and semantic segmentation result; Among them, the lightweight semantic understanding model is trained based on images of key parts of historical targets, including a lightweight backbone network that uses a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature maps extracted by the lightweight backbone network in parallel.

[0059] Among them, images of key parts of historical targets include network images and rendered images; The steps to obtain the rendered image include: Collect 3D models of each target category from the Internet; The 3D model is processed by changing the camera's perspective, position, distance, and background to obtain a rendered image.

[0060] Specifically, the lightweight backbone network performs feature extraction on the target key part image and outputs multi-scale feature maps; the scales of the feature maps are 1 / 8, 1 / 16 and 1 / 32 of the target key part image respectively.

[0061] The object detection head includes an anchor frame layer, a detection layer, and a prediction layer; The anchor frame layer is used to adapt to objects of different sizes in the feature map according to preset anchor frames of different proportions and sizes, so as to generate candidate frames for objects of different sizes on the feature map; The detection layer includes a convolution layer for processing feature maps and extracting features, and can adapt to different detection tasks by adjusting the number of channels and the kernel size; The prediction layer includes a convolutional layer and an output layer, which is used to predict the bounding boxes and categories of different targets based on the extracted features; wherein the number of bounding boxes and the number of categories predicted are determined by the number of channels of the output layer.

[0062] Here, a non-maximum suppression algorithm can be used to suppress the output bounding box to retain the most representative bounding box.

[0063] The semantic segmentation model adopts the PSPNet network model, and the PSPNet network model is additionally provided with a pyramid pooling module for enhancing the ability of the semantic segmentation model to capture details and semantic information simultaneously.

[0064] Here, in the loss function design process of the lightweight semantic understanding model, the weight of the target detection loss and the weight of the semantic segmentation loss are adjusted to control the proportion of the two in training. The specific formula is as follows:

[0065] in, Represents the total loss of the lightweight semantic understanding model; represents semantic segmentation loss; represents the target detection loss; represents the semantic segmentation loss ratio; Represents the target detection loss ratio.

[0066] like Fig.23 As shown, the present invention also provides a lightweight semantic understanding system for target key parts, including: an image acquisition module, used to acquire target key part images; a semantic understanding module, used to input the target key part images into a pre-trained lightweight semantic understanding model, and output target detection results and semantic segmentation results; wherein the lightweight semantic understanding model is trained based on historical target key part images, including a lightweight backbone network using a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature map extracted by the lightweight backbone network in parallel.

[0067] The present invention also provides a device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the lightweight semantic understanding method for a target key part when executing the computer program.

[0068] When the processor executes the computer program, the above-mentioned steps of lightweight semantic understanding for the key parts of the target are implemented, for example: collecting images of the key parts of the target; inputting the images of the key parts of the target into a pre-trained lightweight semantic understanding model, and outputting target detection results and semantic segmentation results; wherein the lightweight semantic understanding model is trained based on historical images of the key parts of the target, including a lightweight backbone network that uses a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature map extracted by the lightweight backbone network in parallel.

[0069] Alternatively, when the processor executes the computer program, it implements the functions of each module in the above system, for example: an image acquisition module, used to acquire images of key parts of the target; a semantic understanding module, used to input the images of key parts of the target into a pre-trained lightweight semantic understanding model, and output target detection results and semantic segmentation results; wherein the lightweight semantic understanding model is trained based on historical images of key parts of the target, including a lightweight backbone network that uses a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature map extracted by the lightweight backbone network in parallel.

[0070] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete preset functions, and the instruction segments are used to describe the execution process of the computer program in the lightweight semantic understanding device for the key parts of the target. For example, the computer program can be divided into an image acquisition module and a semantic understanding module; the specific functions of each module are as follows: an image acquisition module for acquiring images of key parts of the target; a semantic understanding module for inputting images of key parts of the target into a pre-trained lightweight semantic understanding model, and outputting target detection results and semantic segmentation results; wherein the lightweight semantic understanding model is trained based on historical target key parts images, including a lightweight backbone network using a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature map extracted by the lightweight backbone network in parallel.

[0071] The lightweight semantic understanding device for the target key parts can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The lightweight semantic understanding device for the target key parts may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above is an example of a lightweight semantic understanding device for the target key parts, and does not constitute a limitation on the lightweight semantic understanding device for the target key parts. It may include more components than the above, or a combination of certain components, or different components. For example, the lightweight semantic understanding device for the target key parts may also include input and output devices, network access devices, buses, etc.

[0072] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the lightweight semantic understanding for the key target parts, and uses various interfaces and lines to connect the various parts of the entire lightweight semantic understanding device for the key target parts.

[0073] The memory can be used to store the computer program and / or module, and the processor implements various functions of the lightweight semantic understanding device for target key parts by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.

[0074] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0075] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the lightweight semantic understanding method for key parts of a target are implemented.

[0076] The module / unit integrated by the lightweight semantic understanding system for target key parts can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product.

[0077] Based on such understanding, the present invention implements all or part of the processes in the above-mentioned lightweight semantic understanding method for target key parts, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of the above-mentioned lightweight semantic understanding method for target key parts can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.

[0078] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0079] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0080] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention.

Claims

1. A lightweight semantic understanding method for key target parts, characterized in that: include: Collect images of key parts of the target; Input the target key part image into the pre-trained lightweight semantic understanding model, and output the target detection result and semantic segmentation result; Among them, the lightweight semantic understanding model is trained based on images of key parts of historical targets, including a lightweight backbone network that uses a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature maps extracted by the lightweight backbone network in parallel.

2. The lightweight semantic understanding method for target key parts according to claim 1 is characterized in that: The images of key parts of the historical targets include network images and rendered images; The steps to obtain the rendered image include: Obtain 3D models corresponding to various targets; The 3D model is processed by changing the camera's perspective, position, distance, and background to obtain a rendered image.

3. The lightweight semantic understanding method for target key parts according to claim 1 is characterized in that: The lightweight backbone network performs feature extraction processing on the target key part image and outputs a multi-scale feature map; wherein the scales of the feature map are 1 / 8, 1 / 16 and 1 / 32 of the target key part image respectively.

4. The lightweight semantic understanding method for key target parts according to claim 1 is characterized in that: The target detection head includes an anchor frame layer, a detection layer and a prediction layer; The anchor frame layer is used to adapt to objects of different sizes in the feature map according to preset anchor frames of different proportions and sizes, so as to generate candidate frames for objects of different sizes on the feature map; The detection layer includes a convolution layer for processing feature maps and extracting features, and can adapt to different detection tasks by adjusting the number of channels and the kernel size; The prediction layer includes a convolutional layer and an output layer, which is used to predict the bounding boxes and categories of different targets based on the extracted features; wherein the number of bounding boxes and the number of categories predicted are determined by the number of channels of the output layer.

5. The lightweight semantic understanding method for target key parts according to claim 4 is characterized in that: The output bounding box is suppressed using the non-maximum suppression algorithm to retain the most representative bounding box.

6. The lightweight semantic understanding method for target key parts according to claim 1 is characterized in that: The semantic segmentation model adopts the PSPNet network model, and PSPNet introduces a pyramid pooling module for fusing context information of different scales.

7. The lightweight semantic understanding method for target key parts according to claim 1 is characterized in that: In the loss function design process of the lightweight semantic understanding model, the weight of the target detection loss and the weight of the semantic segmentation loss are adjusted to control the proportion of the two in training. The specific formula is as follows: in, represents the total loss of the lightweight semantic understanding model; represents semantic segmentation loss; represents the target detection loss; represents the semantic segmentation loss ratio; Represents the target detection loss ratio.

8. A lightweight semantic understanding system for a key part of a target, used to implement the steps of the lightweight semantic understanding method for a key part of a target as described in any one of claims 1 to 7, characterized in that: include: An image acquisition module, used to acquire images of key parts of the target; The semantic understanding module is used to input the target key part image into the pre-trained lightweight semantic understanding model and output the target detection result and semantic segmentation result; Among them, the lightweight semantic understanding model is trained based on images of key parts of historical targets, including a lightweight backbone network that uses a feature pyramid network as an image processing entrance, a target detection head for identifying bounding boxes and categories of different targets in an image, and a semantic segmentation model for identifying the category to which pixels in an image belong; the target detection head and the semantic segmentation model process the feature maps extracted by the lightweight backbone network in parallel.

9. A device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the lightweight semantic understanding method for key parts of a target as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the steps of the lightweight semantic understanding method for the key parts of the target as described in any one of claims 1-7.

Images