Image annotation method and device, image semantic segmentation method and device and model training method and device

[0085]In order to better understand the technical solutions of the present application embodiments, the following will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0086]In order to facilitate the understanding of the technical solutions of the present application, the nouns of the present application will be described below.

[0087]1. Artificial Intelligence (AI): It is the utilization of digital computer or digital computer-controlled machine simulation, extension, and expanded people's intelligence, perceptual environment, acquisition knowledge and use knowledge to obtain the best results. system. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar way. Artificial intelligence is to study the design principles and implementation methods of a variety of intelligent machines, making machines with perception, reasoning and decision-making.

[0088]Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, and there are both hardware-level technologies. Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large data processing technology, operation / interactive system, and electromechanical integration. Artificial intelligence software technology mainly includes computer visual technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0089]2, machine learning (Machine Learning, ML): It is a multi-mate study in a multi-field cross-discipline, involving probability, statistical, approximation, convex analysis, algorithm complexity theory, etc. Specializing in how to simulate or achieve human learning behavior, obtain new knowledge or skills, reorganizing existing knowledge structure constantly improving its performance. Machine learning is the core of artificial intelligence. It is the fundamental way to make the computer intelligence, which is used throughout the field of artificial intelligence. Machine learning and depth learning usually include artificial neural networks, confidence networks, strengthen learning, migration learning, induction, study, teaching learning, etc.

[0090]3, convolutional neural network (CNN): is a feedforward neural network, which can respond to a part of the coverage, which has excellent performance for large image processing. The convolutional neural network consists of one or more convolutional layers and the top of the entire layer (corresponding to the classic neural network), and also includes association weight and poolizing layer.

[0091]4. Depth Features: The image features extracted by the depth network include abstract information of the image.

[0092]5, semantic segmentation: assign the corresponding category tag based on the object belonging to each pixel in the image.

[0093]6, semantic image: Assign the results obtained after each pixel in the image.

[0094]7. Mask Image: In the present application embodiment, an image used to represent an image block selected in an image, a mask image, for example, a binary image, a binary image including a first value of a first value, a first type of pixel point and a pixel The value is a second type of pixel point, such as a value of one pixel point in a binary image, meaning that the pixel point is not selected, if the value of the pixel point in the binary image is 1, means it The pixel point is selected.

[0095]8, Conditional Generative Adversarial Nets, CGAN): An improvement made on GAN basis, and add additional conditional information to the original GaN generator generator and the discriminator to achieve the conditional generating model. Additional condition information can be a category tag or other auxiliary information.

[0096]9. ImageNet Database: The large-scale database containing 1000 categories.

[0097]10, Mobilenetv2: A commonly used lightweight network model architecture, training on the ImageNet database, can be used to extract image features.

[0098]11, image classification and category: Image classification refers to image processing methods that separate different categories of targets according to different features reflected in image information. It utilizes a computer to quantify the image, and simultaneously divide each pixel point or area in the image or image into some of several categories, instead of the visual judgment of the person. Categories can also be called classification. The category of the present application may have two or more, such as vehicles, highways, and the like. When the image semantic segmentation model is applied to different scenes, the categories that need to be marked can be different. Each target in the image is actually constructed by the pixel point, the category of the so-called pixel is the category of the target.

[0099]12, sample image and target image: all belong to an image, and the image of the training model is referred to as a sample image in the present application embodiment, and an image processed by the subsequent model is referred to as a target image.

[0100]13, artificial intelligence game role: refers to the game role that uses artificial intelligence technology control in the game, including non-Player Character, NPC), or player role in a specific case, can also be referred to as artificial intelligence Game roles, for example, when the player does not control the player role within a preset, the manual intelligence technology can be used to control the player role to perform game tasks.

[0101]14, edge information and edge pixel point: edge information is used to describe information of pixel points in image of pixel point in the image, the pixel point of the pixel point change, these pixel points, the neighborhood pixel point grayscale change That is, the edge pixel point, the edge information may include the gradation value of each edge pixel point, and a shape composed of each marginality or the like. The edges are widely present between the objects and the background, and the objects are between the objects. Edge information in the image can be obtained by image edge detection.

[0102]In order to increase the annotation efficiency of the sample image, the present application provides an image labeling scheme that acquires the edge information of the sample image, and selects a partial image block from the sample image based on edge information, and makes a category label for the selected partial image block. In this way, it can reduce the quantity, improve labeling efficiency, and reduce human cost. At the same time, the selected partial image block includes an image block having a rich edge feature, and the edge feature is rich in margin. It has a large possibility of the category information, so the screening of edge pixel points with edge features satisfies the image block of certain conditions. Make more types of image blocks that have more categories, which will not affect the accuracy of model training. In addition, since the sample image is not labeled, the randomness of the sample image after the label is increased, and the training of the model after training can avoid the extension of the model.

[0103]Based on the above design ideas, the application scenarios of the image labeling method according to the present application embodiment will be described below.

[0104]The sample image after reference in the present application embodiment can be used to train the image semantic segmentation model, and the image semantic segmentation model can output the category of each pixel point in the image, so the image labeling method in the present application can be applied to any needs. The scene of the image labeled, for example, in the game scene, specifically, according to the image segmentation result generated by the image semantic segmentation model, control the artificial intelligent game role, for example in the gun game, the image content is parsed by the semantic segmentation model, thus Providing location information of important targets such as housing, vehicles, artificial smart game characters can perform housing exploration and driving carrier and other game tasks based on these location information. For example, the image labeling method in the present application embodiment can also be applied to the automatic driving scenario, and specifically, according to the image segmentation result generated by the image semantic segmentation model, the position information of the important target is determined, thereby providing a reference for vehicle travel.

[0105]Please refer tofigure 1 The application scenario application for the application of the image label method of the present application embodiment includes a plurality of servers and terminals 140.

[0106]Multiple servers include first server 110, second server 120, and third server 130, first server 110 for implementing sample image labels. The second server 120 is used to acquire the sample image after the first server 110, and based on the labeling sample image, training the image semantic segmentation model. The third server 130 is configured to obtain a trained image semantic segmentation model from the second server 120, and provide image semantic segmentation function using the training-trained image semantic segmentation, and the terminal 140 and the third server 130 can communicate with each other. Use the terminal of the image semantic segmentation function. Among them, the image labeling method, model training method, and image semantic segmentation method will be described below.

[0107]It should be noted,figure 1 In the case of labeled sample images, training models, and implementation of image semantics through server implementation, it can actually achieve corresponding functions through the terminal. In addition,figure 1 The implementation of three different devices are achieved by labeled sample images, training models, and implementation of image semantics. It can actually be implemented by one or two devices, and this application does not limit this.

[0108]Further, between the terminal 140 and the third server 130 can be directly or indirectly connected by a wired or wireless communication, and this application does not limit this. Additionally, the terminal 140 can also be installed with the client 141, and the client 141 is communicating with the third server 130 to implement the corresponding image semantic segmentation function.

[0109]For example, client 141 is a gaming client, and the third server 130 can control artificial smart game characters to perform the corresponding task based on the trained image semantic segmentation model, and update the game screen in real time, and send the updated game screen to the terminal. 140. The terminal 140 is received and presented. Alternatively, for example, the third server can utilize the control artificial smart game role, test the game application, and send the test results to the terminal 140.

[0110]Among them, the terminals in the above may be smartphones, tablets, laptops, desktop computers, smart speakers, smart watches, game devices, smart TV, smart bracelets, etc., but are not limited to this. The first server 110, the second server 120, and the third server 130 can be a stand-alone physical server, or a server cluster or distributed system constructed of a plurality of physical servers, but also providing cloud services, cloud databases, cloud computing. , Cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (Content DeliveryNetwork, CDN), and cloud servers for infrastructure cloud computing services such as big data and artificial intelligence platforms.

[0111]On the basis of the application scenarius discussed above, the image labeling method according to the first server is used as an example, and the image labeling method according to the embodiment is described.

[0112]Please refer tofigure 2 The flow chart of an image labet method provided in the present application embodiment includes:

[0113]S201, the first server divides the sample image to be labeled into multiple image blocks.

[0114]The first server can obtain one or more sample images, the processing of each sample image is the same, and in the present application embodiment, the process of labeling a sample image is an example, a process of labeling sample images.

[0115]After the first server obtains the sample image, the sample image can be divided into a plurality of image blocks, and the first server can divide the sample image into a plurality of magnitude image blocks in accordance with the fixed size, or the first server can be fixed by fixed quantity. The sample image is divided into a fixed number of plurality of image blocks, wherein the size of any two image blocks may be the same, or may be different, and the present application does not limit the specific method of dividing the sample image.

[0116]S202, the first server determines an edge pixel point having an edge feature in a plurality of image blocks, respectively.

[0117]The first server can detect the edge information of the sample image, determine an edge pixel point having an edge feature in the sample image according to edge information, and the corresponding edge pixel point included in each image block is determined. The edge pixel point can be understood as a pixel point that changes in the pixel value.

[0118]S203, the first server from the plurality of image blocks, screening the number of edge pixel points satisfying at least one target image block of the preset screening condition.

[0119]The first server can screen one or more target image blocks from the plurality of image blocks according to the number of edge pixel points included in each image block. For example, the first server can screen the number of image blocks in the number of edge pixel points as the target image block from the plurality of image blocks.

[0120]S204, the first server is labeled for at least one target image block to obtain the labeling result of the sample image.

[0121]After obtaining the at least one target image block, the first server can be labeled with at least one target image of the screen. For example, the first server can be a category result of obtaining each pixel point in each of the target image blocks according to the user's input information. Alternatively, for example, the first server can automatically recognize the category results of all pixel points in each target image block, thereby obtaining the labeling result of the sample image.

[0122]The first server can train the image semantic segmentation model according to the label results, or send the labeling result and the sample image to other devices, so that other devices can train image semantic segmentation models based on sample images and label results.

[0123]In the present application embodiment, the edge information in the sample image can be detected, generating an image edge, and select the target image block that requires the margin based on the edge information, and subsequently labeled the target image block, which reduces the label of each sample image. Quantity, improved annotation efficiency. Further, since the edges of the edge are likely to be different types of objects, the target image block is selected based on edge information, and the target image block containing the number of categories can be selected, and the accuracy of the subsequent training model can be selected. Accuracy.

[0124]Belowfigure 2 Based on the embodiment, the specific embodiment of each step is described:

[0125]The first server needs to obtain the sample image before executing the sample image, and the first server gets the sample image mode:

[0126]Way, the first server can screen itself from the network as a sample image.

[0127]The first server can be based on the screening rules, and the desired sample images required from the network can be filtered, such as one or two of the image quality screening rules, image scenario rules. The image quality screening refers to the image quality, the image quality can include one or two of the image definition and color saturation, and the image scene screening refers to the scene according to the image, and screens the image that meets the target scene. . The target scene can be an application scenario of an image semantic segmentation model, for example, an image semantic segmentation model is applied to the game scene, then the first server can screen from the network as a sample image as a sample image.

[0128]Method 2: The first server can obtain a sample image from other devices.

[0129]Specifically, the first server can obtain a sample image from the application scenario-related device, such as an image semantic division model, such as an image semantic division model, such as an image semantic division model, such as an image semantic division model, such as an image semantic segmentation model. Get sample images.

[0130]Method 3, the first server combines the above-described way one and mode two, obtain the sample image.

[0131]Regardless of the first server to acquire a sample image, the first server can be obtained directly to sample video, and thus, in the present application embodiment, the first server can screen sample images from the sample video.

[0132]For example, the first server randomly samples sample video to obtain a sample image.

[0133]Alternatively, the first server can sample the sample video according to the preset sampling interval to obtain a plurality of candidate images, and the first server can directly obtain a sample image as a sample image, which can simply quickly obtain a sample image.

[0134]In order to improve the effectiveness of the sample image, the first server can screen off a candidate image having a higher similarity in a plurality of candidate images to obtain a sample image.

[0135]Specifically, after obtaining a plurality of sample images, the first server can determine the similarity between each of the two sample images, such as each of the respective image feature vectors of two sample images, calculating two image feature vectors. Similarity, the similarity of two image feature vectors can be characterized by cosine similarity or European distance between two image feature vectors. After obtaining the similarity between the two sample images, if there is any two candidate images of the preset similarity, one of the candidate images can be kicked, and the elimination can be understood as deleting the candidate image. As a sample image, the remaining candidate image is pushed as a sample image to be labeled. A preset similarity is a preset similarity threshold, and the specific value can be set according to the requirements, for example, the value is 0.9.

[0136]In the present application embodiment, a candidate image having a higher similarity can be screened, and a case where the model is prefed from the model to the model to be trained with a candidate image of a similar degree can be avoided.

[0137]For example, the preset similarity is 0.9, and the candidate image includes the similarity between images A, B, D, D, and the similarity between the A and B of 0.95, B, and C is 0.2, C and D. The similarity between the similarity of 0.91, A and D is 0.3, and the first server determines that the similarity between A and D is greater than the similarity of the preset similarity, C and D is greater than the preset similarity. The first server can eliminate A and C, and the remaining B and D are sample images.

[0138]After obtaining the sample image, the first server can divide the sample image to obtain a plurality of image blocks, and the division can refer to the contents discussed above, and will not be described here. After obtaining a plurality of image blocks, the process of S202 is executed, and the implementation of the execution S202 is executed below:

[0139]The first server can perform grayscale processing of the sample image to obtain a sample image after graylation, and then extract the gradation sample image to perform an edge detection process, obtain a plurality of image blocks having an edge feature in a plurality of image blocks.

[0140]For example, please refer toFigure 3A An example of a graphic treatment for grayscale processing for a sample image.

[0141]Specifically, the first server can convert the sample image to a grayscale, and then extract the edge information of the image based on the preset edge detection algorithm, preset edge detection algorithm, such as a CANNY edge detection algorithm, and the CANNY edge detection algorithm as an example to the edge Example of the testing process:

[0142]S1.1: Gaussian filtering of the sample image after grayscale.

[0143]The main purpose of Gaussian filter is to reduce the noise of sample images after grayscale. The gray-grade sample image can actually be understood as a weighting average of the sample image after grayscale, that is, the gradation value of each pixel point in the sample image after grayscale The gradation value of other pixel points in the neighborhood of the pixel point and the pixel point is obtained after weighted average. Gaussian filtering is weighted by the grayscale value of each pixel point in the sample image after graylation, thereby filtering out some of the noise in the image, so that the overall contour in the gradation sample image is relatively blurred, so that after processing The overall image is smoother, and the width of the contour is increased.

[0144]ContinueFigure 3A The example shown,Figure 3A After the sample image shown, the Gaussian filter is performed.Figure 3B The example diagram shown,Figure 3B Compared toFigure 3A As shown, the integral line of the image is smoother.

[0145]S1.2: Calculate the gradient value and gradient direction in the image after Gaussian filtering.

[0146]The edges can be understood as a collection of pixel points with large change in grayscale values, such as a black edge, a white edge, then part of the black and white side is generally the edge, and when implementation, grayscale can be detected. The value change, thereby identifying the edges in the image in which the gradient value can be used to represent the degree of change in the gradation value, and the gradation direction is indicated in the gradation direction. Where the gradient value and gradient direction can be calculated by the following formula:

[0147]

[0148]

[0149]Among them, gxIndicates the gradation value obtained by the lateral edge detection, gyIndicates that the gradation value obtained by the longitudinal edge detection, G is the degree of change of the gradation value, θ represents the gradient direction.

[0150]S1.3: Filtering the Nonestitus.

[0151]In S1.1, since the contour width in the image is actually amplified, this may affect the accuracy of the detection edge, so S1.3 is mainly used to sieve the pixel point that is not the edge.

[0152]Specifically, if the first server determines that the gradient value in the gradient point in the gradient direction is determined that the pixel point belongs to the suspected edge pixel point; if it is determined that the gradient value in the gradient direction is not the largest, then determined The pixel point is not an edge pixel point, so that it excludes some pixel points that are not endless. Suspected edge pixel points can be understood as preliminary identification as an edge pixel point, but can be further determined.

[0153]In a possible embodiment, the first server can directly use the suspected edge pixel point as an edge pixel point, thereby obtaining an edge pixel point in a sample image after graylation.

[0154]S1.4: Determine the edge using the upper threshold.

[0155]In order to determine a more accurate edge pixel point, the first server can also be screened in S1.3 in the suspected edge pixel point in S1.3 in the first server in the present application embodiment. Among them, the high threshold is greater than the low threshold.

[0156]Specifically, if the first server determines that the gradient value of the suspected edge pixel is greater than the high threshold, it is determined that the suspected edge pixel point is the edge pixel point; if it is determined that the doubt is less than the high threshold but is greater than the low threshold, it is determined. The suspected edge pixel point belongs to the edge pixel point; if it is determined that the gradient value of the suspected edge pixel is less than or equal to the low threshold, it is determined that the suspected edge pixel point is not the edge pixel point. In this way, the first server can determine all edge pixel points in the sample image after grayscale, so that an edge image can be obtained. The edge image can be understood as an image identifically an edge pixel point and a non-edge pixel point. Non-edge pixel points can be understood as a pixel point that does not belong to the edge pixel point in the image.

[0157]After obtaining an edge image, the first server further determines the edge pixel point according to the gradation value of the edge pixel point, such as the gradation value of the pixel point is the edge pixel point of the preset value. After obtaining an edge image, the first server can naturally determine the number of edge pixel points included in each image block. It should be noted that the number of edge pixel points included in each image block may be 0, 1, or more.

[0158]ContinueFigures 3A ~ 3B The example shown, the first server processes the sample image, obtainedFigure 3C Edge image shown,Figure 3C The black lines correspond to the edge.

[0159]After determining the number of edge pixel points included in each image block, S203 can be performed, and the first server can determine the target image block from a plurality of image blocks, and the specific embodiment of S203 is described below:

[0160]Specifically, the first server can determine the edge pixel point included in each image block, calculate the ratio of the edge pixel points included in each image block and all edge pixel points in the sample image, thereby based on the ratio of each image block. Determine at least one target image block.

[0161]The following describes the formula of the calculated ratio:

[0162]

[0163]Where PiIndicates the ratio corresponding to the first image block, represents the number of edge pixel points included in the i-th grid, D represents the number of plurality of image blocks included in the sample image, such as the sample image divided 16 image blocks, then D value of D is 16, and n will take D.

[0164]After the first server determines the ratio corresponding to each image block, at least one target image block can be determined according to the ratio, there are a variety of methods, and the example is described below:

[0165]Determine the method 1. The ratio corresponding to the plurality of image blocks is determined from large to small, and the image block corresponding to the previous N ratio is determined as the target image block.

[0166]After the first server determines the ratio of the plurality of image blocks, the plurality of ratios can be sorted from a small, thereby obtaining a plurality of ratios after the sorting, and determines the N rattles before the sorting multiple ratios. The image block corresponding to this N ratio is determined as the target image block.

[0167]In the present application embodiment, the image block is determined from the plurality of image blocks as the target image, and the determination method is relatively simple, and the image block is large due to the larger than the larger value indicates that the edge pixel point in the image block is large, and the edge is two The side is more likely to be a different category of objects, the more edge pixel points indicate relatively more in the image block, so that the category information can be obtained more enriched image blocks to facilitate subsequent training image semantic segmentation models.

[0168]Where n is a preset natural number. As an embodiment, n can be set according to the number of image blocks included in the sample image, such as having a number of plurality of image blocks, which can make the number of target image blocks relatively reasonable.

[0169]Determine method 2, determine the image block of the ratio of not less than the preset ratio as the target image block.

[0170]The first server can be provided with a preset ratio, which can be set according to the actual situation of the sample image, the first server after determining the ratio of each image block in the plurality of image blocks, can increase the ratio greater than or equal to the preset ratio. The image block is determined as a target image block.

[0171]In the present application embodiment, the target image block is determined in the predetermined ratio, and when the edge pixel point of the target image block is guaranteed, the number of target image blocks can be made higher. , The image block containing the edge pixel point is kept as much as possible.

[0172]Determine the method three, with the ratio of the plurality of image blocks as the random probability, randomly select at least one target image block from the plurality of image blocks.

[0173]The first server can use the ratio of each image block as the random probability of the image block, and then, according to the random probability of respective corresponding to the plurality of image blocks, at least one target image block is randomly outward from the plurality of image blocks. When at least one target image block is random, the first server can set a predetermined amount to randomly out of the preset number of at least one target image block.

[0174]In the present application embodiment, the first server is a random probability of a ratio, which is randomly out of the target image block in a plurality of image blocks, which can be randomly out of the image block of the edge pixel point, but not Directly selects more image blocks in edge pixel, so that the screening of the target image block has certain randomness, which facilitates the avoidance of the prefraction of the late image semantic segmentation model.

[0175]After obtaining at least one target image block, in order to facilitate the subsequent determination of which is a target image block, which is no selected image block, in the present application embodiment, the first server can be labeled for the target image block in the sample image. In order to be subsequently identified which is the target image block.

[0176]Specifically, the first server can be labeled on at least one target image block, and it is understood that each pixel point in each of the target image blocks has a first identifier in each of the target image blocks in the at least one target image block, and multiple images In addition to other image blocks other than the at least one target image block, the block is labeled a second identifier to obtain a mask image.

[0177]The first identification and the second identity belong to different identifiers, the first identification and the second identifier may belong to the same type, but it belongs to the different identifiers under the same type, such as the first identification and the second identification, and the color representation, for example, for example The first logo is white, and the white specific may "1" indicates that the second identification is black, and the black specifically can be represented in "0". The first identification and the second identification belong to the same type, and it is convenient for subsequent devices to resolve the mask image. The first identification and the second identification can also belong to a different type, and this application does not limit this.

[0178]For example, continue to useFigure 3A The example shown, the first server determines the target image block in the sample image, and the first server is labeled for the sample image, thereby obtainingFigure 4A The mask image shown in, wherein the target image block is labeled as white, and other image blocks other than the target image block are labeled black in the plurality of image blocks.

[0179]After determining the target image block, the first server can perform S204, the following uses the first server as an example, and performs an example of the process of category the category of at least one target image block:

[0180]The first server can obtain a category of each pixel point in a target image block according to the user's input information, which includes a category corresponding to each pixel point, such as a labeling operation of the type of each pixel point, and the first server according to the Task operations, get input information, thereby obtaining the categories of each pixel point. Alternatively, the first server can automatically identify the category of each pixel point, labeled each pixel point, such as the first server, according to the edge information of the sample image, matches the various targets in the sample image according to edge information, thereby performing various pixel points Label.

[0181]Among them, the tags of the label can be two or more, the category of the label can be the first server set, for example, a bridge, people, grass, trees, houses, backgrounds, doors and windows, etc.

[0182]Regardless of the first server, the first server is marked with each pixel point, there are a variety of ways that belong to different categories of pixel points, such as the first server can be different from different categories of pixels in different colors. Marking, you can also mark different categories of pixel points on different colors, or different types of pixel points can be labeled for different transparency. Or can also perform different categories of pixel points in different numerals. Note, etc., the present application is not limited.

[0183]For example, continue to useFigure 3A The example shown, the first server is filtered outFigure 3A After the target image block in the sample image shown after the grayscale, the first server can select the target image block to be labeled, thereby obtainingFigure 4A The mask image shown,Figure 4A The white mask image corresponds to the target image block. Further, the first server can be labeled in each pixel point in the target image block, specifically a serial number corresponding to the category of different categories of pixel points, which is obtained.Figure 4b Please refer to the result of the labelingFigure 4b Where the number of pixels corresponding to the serial number 1 belongs to the bridge, the corresponding pixel point belongs to the vehicle, and the number of pixels 3 belongs to people. CombineFigure 4b It can be seen that the method in the present application embodiment can not need to be labeled over the entire image, thereby reducing the quantity.

[0184]The above is an example introduction of the process of processing a sample image, but when actually implemented, the first server can perform the sample image labeling process of each sample image in a plurality of sample images to obtain each sample image. Labeling results, these sample images can be used to be used for training of image semantic segmentation models.

[0185]infigure 2 In the illustrated embodiment, when the sample image is labeled, it is not marked for each pixel point in the sample image, but the sample image is divided into a plurality of image blocks, and then the edge information is more abundant. Block, labeled these selected target image blocks, which can reduce the amount of labeling during the labeling sample image. Moreover, since the screening is the image block, the edge information is richering to indicate that the category information contained in the image block is richer, which can be reduced, and the information of the image block in the sample image itself has a certain amount. Redundancy, so even if you do not mark all image blocks, it will not affect the accuracy of the post-image semantic segmentation model training. Further, since the selected image block may not contain a complete target, it can avoid the over-fitting training of the image semantic segmentation model, and the optimization ability of the image semantic segmentation model can be improved.

[0186]Further, after experimentation verification, the present application decreases the amount of more than 50% or more, and does not reduce the effect of image segmentation.

[0187]In order to more clearly explain the image labeling method in the present application embodiment, the following is applied to the gun battlefield scene in the game scene in the image semantic division model as an example, and an example introduction of the image labeling method in the present application embodiment.

[0188]S2.1, the first server gets the gun battle game image.

[0189]Get the sample video corresponding to the gun battle game, from the sample video to collect the marked gun game image, the frequency of the sampling is 5 seconds, so the purpose is to avoid the similaritten between the images.

[0190]Further, after acquiring the game image of the gun, it is possible to screen the similaritten high gun game image, which in turn obtain the sample image, and the sample image can specifically 5,000 guns game images. The way to filter out similar to high gun game images can refer to the contents discussed foregoing, and will not be described again here.

[0191]S2.2, the first server extracts the image edge.

[0192]After obtaining the gun game image, the first server can be preprocessed to the gun battle game image. Specifically, the edge information of the extracted gun game image, the edge information can be understood as the contour information of the object, because the edges of the edge are usually different categories. Target, the edge information can assist in subsequent selection of image blocks from the gun game image.

[0193]Specifically, the first server can gray the gun game image, extracts the edge of the gun battle game image after the graylation of the gamma based on the Canny edge detection algorithm, thereby obtaining an edge image, and the pixel value of the edge of the object in the edge image is 1, Non-edge corresponding pixel values ​​are 0.

[0194]S2.3, the first server determines the target image block.

[0195]After the first server obtains an edge image, the gun battle game image can be divided into a plurality of image blocks, such as dividing an image block of 4 × 4, i.e., is divided into 16 image blocks, and calculate the edge pixel point included in each image block. Number, the edge pixel point is specifically the edge point is a pixel point of the pixel value of 1 in the edge image. The first server then calculates the random probability of the image block is calculated according to the number of each image block edge pixel point point, and the calculation of the random probability can be referred to the contents discussed foregoing, and will not be described again here.

[0196]The first server can select the target image block according to the random probability, such as the first server can select 8 image blocks from 16 image blocks.

[0197]In one possible embodiment, the first server generates a binarized mask image.

[0198]Specifically, after obtaining the target image block, each pixel point can be labeled a first value in the target image block, and other image blocks in the plurality of image blocks are second values, generating a mask image.

[0199]S2.4, labeled the target image block.

[0200]The first server can label each pixel of the objective input target image block, thereby labeled each pixel point in the target image block, and the category of the mark is required: people, grass, trees, houses, backgrounds, doors ,window. For example, eight target image blocks are filtered from 16 image blocks, one, only need to target 8 target image blocks, reduce 50% labeled quantity, reduce the human cost of labeling.

[0201]Based on the above image labeling method, the present application provides an image semantic division model training method, and the method is performed by the second server as an example, and combinedFigure 5 A flowchart of an image semantic segmentation model training method is introduced:

[0202]S501, the second server obtains the labeling result of the sample image.

[0203]The second server can obtain a sample image from the first server, and the labeling result of the sample image, and a mask image or the like can be obtained. Alternatively, the second server can also obtain the labeling result of the sample image through the image labeling method of the previous discussion. The specific process of obtaining the results of the sample image can refer to the contents discussed foregoing, and details will not be described herein.

[0204]It should be noted that the second server can obtain a labeling result of each sample image in a plurality of sample images in the above-described manner, so that multiple sample images and labeling results of each sample image are trained.

[0205]S502, the second server performs multiple iterative training for the image semantic segmentation model based on the sample image.

[0206]The second server can perform multiple iterative training in the image semantic segmentation model, similar to the process of iterative training, and the process of one iterative training is introduced:

[0207]S3.1 Second server inputs the sample image into the image semantic segmentation model to obtain semantics segmentation results.

[0208]The second server can enter a number of sample images in one iterative training. The number of sample images is, for example, one or more, and the specifically can be a second server set according to the training requirements, a number of sample images that iterately trained. The number can be the same or different.

[0209]When the input of the sample image is multiple, then the image semantic segmentation model can output a semantic segmentation result of each sample image, and the semantic segmentation results corresponding to each sample image include the probability of each pixel point belonging to the various categories in the sample image. . For example, the category includes 7, then the image semantic segmentation model can output the probability of 7 categories per pixel point.

[0210]S3.2, the second server adjusts the model parameters of the image semantic segmentation model based on the results of the labeling, and the semantic segmentation result.

[0211]After obtaining the semantic segmentation result of the sample image, the second server can sequentially divide the semantic semantic semantic semantic semantic semantic semantic point corresponding to the pixel point corresponding to at least one target image block according to the laminated result in the sample image, and the semantic segmentation result. Further, the model parameters of the image semantic segmentation model are adjusted. The second server can determine the semantic segmentation result corresponding to the corresponding pixel point in the sample image, or the second server can obtain a mask image from the first server from the first server, due to the mask image The location information of the target image block is included, so the second server can determine the position of the target image block based on the mask image.

[0212]Among them, the model parameters involving how to adjust the image semantic segmentation model, the following example shows:

[0213]The second server can determine the value of the loss function based on the semantic division of the pixel points corresponding to the pixel points corresponding to at least one target image block in the sample image, and the semantic division of the pixel points corresponding to the at least one target image block, and The value of the loss function adjusts the model parameters of the image semantic segmentation model to reduce the difference between the laminated semantic segmentation results of the image semantic segmentation model.

[0214]Among them, there are a variety of expressions of the loss function, for example, the loss function as shown in the following formula:

[0215]

[0216]Wherein represents the number of sample images of this input image semantic segmentation model, p is the total number of pixel points included in the sample image, C is the total number of categories, mI, P A value corresponding to the pixel point in the mask image corresponding to the i-th sample image, For the probability of the first pyrographic point in the i-th sample image, yI, P, CFor the first pixel point in the i-th sample image, the label results of the category Class C, if the category corresponding to the PP pixel point is C, then YI, P, C1, otherwise YI, P, CThe value is 0.

[0217]S503, until the image semantic segmentation model converges, obtains the trained image semantic segmentation model.

[0218]When the image semantic model is multi-iterative training, if the image semantic split model converges, it is determined that the training is completed and the trained image semantic segmentation model is obtained. The image semantic segmentation model converges may be that the value of the loss function is smaller than the loss threshold, or the number of iterations reaches the number of times, the present application does not limit the specific conditions of the convergence.

[0219]In the present application embodiment, the target image block has a sample image labeled, and the image semantic segmentation model can be trained, and the amount of the sample image can be reduced, and only the partial image block is labeled due to the sample image, therefore calculates the loss. When the function, there is no need to calculate the loss of each pixel point in the sample image, but the loss of each pixel point in the partial image block of the label can reduce the amount of calculation. Further, since all pixel points in the sample image are not labeled in the present application embodiment, the possibility of degrading the image semantic segmentation model is reduced, so that the generalization of the semantic semantic model after training is better.

[0220]As an embodiment, please refer toFigure 6 The image semantic segmentation model includes feature extraction modules and category output modules, which combine belowFigure 6 The model structure shown, the process of outputting a semantic segmentation result of the image semantic segmentation model:

[0221]The feature extraction module is used to extract the depth features of the sample image, and the category output module is used to output the sample image in the probability of each category according to the depth feature.

[0222]Please continue with referenceFigure 6 This feature extraction module can be implemented by the MobileNet network, and the MobileNet network is specifically, for example, MobileNet V2. MobileNet V2 can be pre-trained with images in the ImageNet database. The category output module can be realized by convolutional layers, activation layers, and upper sampling layers, including such asFigure 6 The first activation layer, the first rolling layer, the second activation layer, the second volume layer, the third activation layer, the first upper sampling layer, the fourth activation layer, the second upper sampling layer, the fifth activation Layers, third upper sampling layers, sixth activation layer, fourth upper sampling layer, seventh activation layer, fifth upper sampling layer.

[0223]Specifically, the MobileNet V2 network is used as the feature extraction module, and the convolutionary feature of the sample image is extracted, and the corresponding feature is output, and the feature map is performed by the convolution layer, the output feature is output, and 5 upper sampling layers Expand the scale of the feature. Each up-sampling layer is inserted into a zero point in the middle of the feature map to enlarge the image, and then the enlarged image is convolved, and then the enlarged feature map, for example, the width of the feature can be output. 2 times the feature map. The last sampling layer, its output channel is c, respectively corresponds to the probability of each pixel point belonging to various categories, thereby obtaining each pixel belonging to different categories.

[0224]For example, the second server can separate sample images, label results, and mask images, respectively, to preset size, for example, processed to 640 × 360 × 3, wherein 3 represents the number of channels.

[0225]Enter the processed sample image toFigure 6 The image semantic segmentation model shown is obtained after the MobileNet V2 network processing, the first feature is obtained.

[0226]The first feature layer is sequentially passed through the first activation layer and the first roller layer, the first roll layer of the volume nucleus size of 4, the step length is 2, and the number of output channel is 512.

[0227]Similarly, the second feature is sequentially passed through the second activation layer and the second roll layer, and the third feature of the number of channels is 512. The third feature is input to the third activation layer and the first upper sampling layer to obtain a fourth feature of the channel number 512. The fourth feature is sequentially input to the fourth activation layer and the second sampling layer to obtain a fifth feature of the number of 256. The fifth feature is sequentially input into the fifth activation layer and the third sampling layer to obtain a sixth feature of the channel number 128. The seventh feature is sequentially input to the sixth feature of the sixth activation layer and the fourth sample layer to obtain the number of channels of 64. The eighth feature map is input to the sixth activation layer and the fifth sampling layer to obtain a semantic division image of the channel number 7, which is 640 × 360 × 7 of the semantic division image.

[0228]On the basis of the above-described image semantic division model training method, an image semantic segmentation method is also provided in the present application, and the method is performed by the third server as an example, combined.Figure 7 A flowchart of the image semantic segmentation method shown:

[0229]S701, the third server gets the target image to be divided.

[0230]S702, the third server inputs the target image into the training image semantic segmentation model to obtain the categories belonging to each pixel point in the target image.

[0231]The third server can train the image semantic segmentation model through the previous disciplined image semantic segmentation model training method to obtain the trained image semantic segmentation model. Alternatively, the third server can obtain a trained image semantic segmentation model from the second server. The specific procedure of the training image semantic segmentation model can refer to the contents discussed above, and will not be described herein.

[0232]The third server can input the target image to the image semantic segmentation model, obtain the category belonging to each pixel point in the target image, the image semantic segmentation model can output the probability of each pixel point belonging to each category in multiple categories, third server The category of the probability is the category of the pixel point can be determined.

[0233]In the present application embodiment, the target image can be semantically divided according to the trained image semantic segmentation model. Due to the training image semantic division model, there is no need to mark all the pixel points in the sample image, so the image label is relatively reduced. Further, since the sample image is not fully labeled, the uncertainty of the labeling result of the sample image is improved, so that the processing capability of the image semantic segmentation model can be improved, and the accuracy of the segmentation result obtained by the image semantic segmentation model can be improved.

[0234]Further, the third server can be used in a specific application scenario using the image semantic segmentation model, after obtaining the categories belonging to each pixel point in the target image, the third server can use the categories belonging in each pixel point in the target image to execute Corresponding task.

[0235]For example, the third server can be using the image semantic segmentation model in the game application scenario, then the target image corresponds to the game scene image, the third server after determining the game scene object categories belonging in each pixel point in the game scene image, can be controlled Artificial smart game characters move to the preset game scene category corresponding location, thus controlling artificial intelligent game roles to complete the corresponding task, such as controlling the artificial intelligent game role to perform a house exploration or driving.

[0236]Alternatively, for example, the third server may be using the image semantic division model in the automatic driving application scenario, then the target image corresponds to the traffic route, and the third server can be paired after determining the location of each pixel point in the traffic route. Vehicle conducts navigation.

[0237]Based on the same inventive concept, the present application provides an image labeling device that enables the function of the first server discussed earlier, please refer toFigure 8 The device includes:

[0238]The division module 801 is used to divide the sample image to be labeled into multiple image blocks;

[0239]The module 802 is determined to determine an edge pixel point having an edge feature in a plurality of image blocks, respectively;

[0240]The screening module 803 is configured to screen at least one target image block of the preset screening condition from the plurality of image blocks to the number of edge pixel points.

[0241]The label module 804 is used to mark the at least one target image block, obtain the labeling result of the sample image.

[0242]In one possible embodiment, the device further includes acquiring module 805, acquisition module 805 for:

[0243]The sample video is sampled according to the preset sampling interval before dividing the sample image to be labeled, and a plurality of candidate sample images are obtained.

[0244]In multiple candidate sample images, it is determined that the similarity between any two candidate sample images;

[0245]If there is any two candidate sample images that are greater than the preset similarity, one of the two candidate sample images is eliminated;

[0246]The remaining candidate sample images are sample images to be labeled.

[0247]In one possible embodiment, determining module 802 is specifically used:

[0248]The sample image is grayscale, obtaining a sample image after graylation;

[0249]The edge detection processing after grayscale sample images is obtained, obtaining an edge pixel point having an edge feature in a plurality of image blocks.

[0250]In one possible embodiment, determining module 802 is specifically used:

[0251]Determine as an edge pixel point in a pixel point of the gradation value in the sample image after the grayscale.

[0252]In one possible embodiment, the screening module 803 is specifically used:

[0253]It is determined that the ratio between the edge pixel points included in each image block is between the total number of all edge pixel points of the sample image;

[0254]The at least one target image block is determined from the plurality of image blocks according to the ratio corresponding to each image block.

[0255]In one possible embodiment, the screening module 803 is specifically used:

[0256]The ratio of the plurality of image blocks is sorted from large to small, and the image block corresponding to the previous N ratio is determined as the target image block, and n is a preset natural number; or

[0257]Determine the image block of the ratio of not less than the preset ratio as the target image block; or,

[0258]At least one target image block is randomly selected from the plurality of image blocks with the ratio of the plurality of image blocks.

[0259]In one possible embodiment, the label module 804 is also used in:

[0260]From the plurality of image blocks, the number of out of the edge pixel point satisfies the at least one target image block of the preset screening condition, and the at least one target image block is labeled as the first identifier, and the plurality of image blocks except at least one target image. Other image blocks outside the block are labeled as the second identity, where the mask image is obtained, wherein the first identification and the second identification are different, the mask image is used to train the image semantic segmentation model.

[0261]In one possible embodiment, the sample image is a game scene image having a preset behavior, and the label module 804 is specifically used for:

[0262]Depending on the preset plurality of game scenarios, the game scene belongs to each pixel point in the game scene in the game scene are labeled.

[0263]Based on the same inventory, the present application provides an image semantic division model training device, which can be used to implement the function of the second server discussed above, please refer toFigure 9 The device includes:

[0264]The module 901 is obtained for obtaining the labeling result of the sample image through the previously cultural image labeling method;

[0265]The training module 902 is used to perform multiple iterative training on the image semantic segmentation model according to the sample image;

[0266]The module 903 is obtained for convergence until the image semantic segmentation model is converged, and the trained image semantic segmentation model is obtained.

[0267]The training module 902 is used to perform the following procedure to implement each iterative training in multiple iterative training on the image semantic segmentation model:

[0268]The sample image is input to the image semantic segmentation model, where the semantic segmentation result includes the probability of each pixel point belonging to the various categories in the sample image;

[0269]The model parameters of the image semantic segmentation model are adjusted by the semantic division of the pixel points corresponding to at least one target image block in accordance with the laminated results of the pixel points corresponding to at least one target image block in the sample image.

[0270]Based on the same inventive concept, the present application provides an image semantic division device, which can be used to implement the function of the third server, please refer toFigure 10 The device includes:

[0271]Get module 1001 for obtaining the target image to be divided;

[0272]The module 1002 is obtained for obtaining the trained image semantic segmentation model obtained by the target image input through any image semantic segmentation model training method discussed, and obtains the categories belonging in each pixel point in the target image.

[0273]In one possible embodiment, the target image is a gaming scene image; the apparatus further includes a control module 1003, and the control module 1003 is also used in:

[0274]After obtaining the categories belonging in the target image, the artificial smart game role is controlled to the preset category according to the category belonging to the corresponding task, depending on the category belonging in the target image to perform the corresponding task.

[0275]Based on the same inventive concept, the present application provides a computer device, please refer toFigure 11 The computer device includes processor 1101 and memory 1102.

[0276]Processor 1101 can be a central processing unit (CPU), or a digital processing unit, and the like. Specific connection media between the above memory 1102 and the processor 1101 is not limited in the present application embodiment. The embodiment of this application isFigure 11 The bus 1103 is connected between the memory 1102 and the processor 1101, the bus 1103 isFigure 11 The connection method between the other components is not taken to limit. Bus 1103 can be divided into address bus, data bus, control bus, and the like. For ease of expression,Figure 11 Always use only one thick line, but does not mean that there is only one bus or a type of bus.

[0277]Memory 1102 can be volatile memory, such as random access memory, RAM); memory 1102 can also be non-volatile memory, such as read only memory, flash memory (Solid-State DRIVE, SSD), or Solid-State Drive, SSD, or Memory 1102 is a desired program code that can be used to carry or store the form of instructions or data structures and can be made from a computer. Any other media accessed, but is not limited thereto. Memory 1102 can be a combination of the above memory.

[0278]The processor 1101 performs an image labeling method for performing any one of the previous discussion when the computer program is stored in the memory 1102, and can also be used for implementationFigure 8 The function of the middle device can also be used to implement the function of the first server discussed earlier.

[0279]Based on the same inventive concept, the present application provides a computer device, please refer toFigure 12 The computer device includes processor 1201 and memory 1202. The contents of the processor 1201 and the memory 1202 can refer to the contents discussed foregoing, and will not be described again here.

[0280]The computer program stored in memory 1202 is stored. The processor 1201 is used to invoke a computer program stored in the memory 1202, performing an image semantic division model training method, as previously discussed, and can also be used for implementationFigure 9 The function of the middle device can also be used to implement the function of the second server discussed earlier.

[0281]Based on the same inventive concept, the present application provides a computer device, please refer toFigure 13 The computer device includes processor 1301 and memory 1302. The contents of the processor 1301 and the memory 1302 can refer to the contents discussed foregoing, and details are not described herein again.

[0282]The computer program stored in memory 1302 is stored. The processor 1301 performs an image semantic segmentation method for performing any one of the previous discussion when the computer program is stored in the memory 1302, and can also be used to implementFigure 10 The function of the middle device can also be used to implement the function of the third server discussed above.

[0283]Based on the same inventive concept, the present application provides a computer storage medium, the computer storage medium stores a computer instruction, and when the computer instruction runs on the computer, the computer performs the image labeling method of the computer to perform previous discussion, Image semantic segmentation model training method or image semantic segmentation method.

[0284]Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Accordingly, the present application may employ a full hardware embodiment, a full software embodiment, or in conjunction with embodiments of software and hardware. Moreover, the present application may employ a computer program product that includes a computer available storage medium (including but not limited to disk memory, CD-ROM, optical memory, etc.) implemented in one or more computers.

[0285]Based on the same inventive concept, the present application provides a computer program product, which includes a computer instruction, which is stored in a computer readable storage medium. The processor of the computer device reads the computer instruction from the computer readable storage medium, and the processor executes the computer instruction, so that the computer device performs an image label method, an image semantic division model training method or an image semantic segmentation method of the computer device. .

[0286]One of ordinary skill in the art will appreciate that all or some of the steps that implement the above method embodiments can be done by the hardware associated with the program instruction, and the aforementioned program can be stored in a computer readable storage medium. When executed, execute The steps of the above method embodiments; the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM, ROM, ROM, ROM, RAM, RAM, RAM, RANDOM Access Memory), discharge, or optical discs, etc. The media of the program code can be stored.

[0287]Alternatively, the above-described integrated unit can be stored in a computer-readable storage medium if implemented in the form of a software functional module and is used as a separate product sales or in use. Based on this understanding, the technical solution of the present application example essentially or contributes to the prior art can be embodied in the form of software products, which stores in a storage medium, including several instructions. A computer device (can be a personal computer, server, or network device, etc.) to perform all or portions of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a variety of media such as mobile storage devices, ROM, RAM, disks, or optical discs, can store program code.

[0288]Obviously, those skilled in the art can make various modifications and variations of the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application are within the scope of the claims and their equivalents thereof, the present application is intended to include these modifications and variations.