A scene image analysis method and system

Abstract

The present application relates to a scene image parsing method and system. The main technical scheme includes: collecting multiple scene images as image sample data, labeling the image sample data, and obtaining image label sample data; gridding the image label sample data according to a custom division granularity to obtain grid block sample data; establishing a spatial hierarchical graph structure based on the grid block sample data and extracting feature vector data, assigning the feature vector data to the spatial hierarchical graph structure to obtain feature sample data of the scene image; obtaining a scene image parsing module based on the feature sample data of the scene image and a preset graph model, and constructing a scene image parsing model architecture based on the scene image parsing module; model training the scene image parsing model architecture to obtain a scene image parsing model. The present application can improve the small-scale target feature learning capability of the scene image parsing model and enhance the accuracy of multi-scale scene image parsing.

Description

Technical Field

[0001] The present application relates to the technical fields of computer vision and pattern recognition, and in particular to a scene image analysis method and system. Background Art

[0002] At present, semantic parsing of visual scene images is one of the focuses of computer vision research and is a key technology for achieving semantic-level scene recognition and understanding. Scene parsing requires obtaining multi-label classification results for a given scene image and locating the labels to image pixels to obtain pixel-level semantic parsing results. In traditional implementations, a deep convolutional neural network model is generally used. The model uses densely annotated scene images as training data and uses general convolution, dilated convolution, separable convolution and other computational processes to achieve feature learning.

[0003] However, achieving pixel-level scene parsing based on densely annotated scene images, especially the recognition and segmentation of multi-scale objects in scene images, is still a huge challenge for scene parsing models that directly process raw scene images. Since it is difficult to balance the feature learning weights of larger and smaller objects in scene images, and larger objects can provide more local information, the model tends to learn the features of larger objects, resulting in a decrease in the parsing performance of smaller objects. Summary of the invention

[0004] Based on this, the present application provides a scene image parsing method, system, device and storage medium to effectively implement scene semantic parsing and improve the parsing accuracy of small-scale scene targets.

[0005] In a first aspect, a scene image parsing method is provided, the method comprising:

[0006] Collect multiple scene images as image sample data, label the image sample data, and obtain image label sample data;

[0007] Divide the image label sample data into grids according to the custom division granularity to obtain grid block sample data;

[0008] A spatial hierarchical graph structure is established based on the grid block sample data, and feature vector data is extracted and assigned to the spatial hierarchical graph structure to obtain feature sample data of the scene image;

[0009] A scene image parsing module is obtained according to feature sample data of the scene image and a preset graph model, and a scene image parsing model architecture is constructed based on the scene image parsing module;

[0010] The scene image parsing model architecture is trained based on the image label sample data to obtain a scene image parsing model.

[0011] According to an achievable method in an embodiment of the present application, a spatial hierarchical graph structure is established based on grid block sample data, including:

[0012] Determine the grid category label of the grid block sample data according to the pixel category label of the image label sample data;

[0013] According to the four-neighborhood relationship of the grid block sample data, a spatial edge is established to obtain a spatial graph structure; wherein a graph node represents a grid block sample data;

[0014] The graph nodes with the same grid category labels in each adjacent spatial graph structure under the custom partitioning granularity are connected, and spatial hierarchical edges are established to obtain a spatial hierarchical graph structure.

[0015] According to an achievable method in an embodiment of the present application, extracting feature vector data and assigning the feature vector data to a spatial hierarchical graph structure to obtain feature sample data of a scene image includes:

[0016] The feature extraction algorithm is used to extract features from the grid block sample data to obtain feature vector data;

[0017] The feature vector data is assigned to the corresponding graph nodes in the spatial hierarchical graph structure to obtain the feature sample data of the scene image.

[0018] According to an achievable method in an embodiment of the present application, a scene image parsing module is obtained according to feature sample data of a scene image and a preset graph model, and a scene image parsing model architecture is constructed based on the scene image parsing module, including:

[0019] According to the feature sample data of the scene image and the preset spatial graph convolution formula, a spatial graph convolution operation module of the scene image is obtained;

[0020] The spatial graph convolution operation modules are combined to obtain the graph convolution network unit, and the scene image parsing model architecture is constructed based on the graph convolution network unit.

[0021] According to an achievable method in an embodiment of the present application, a scene image parsing module is obtained according to feature sample data of a scene image and a preset graph model, and a scene image parsing model architecture is constructed based on the scene image parsing module, further comprising:

[0022] According to the feature sample data of the scene image and the preset spectral convolution feature fusion model, a spectral convolution attention module of the scene image is obtained;

[0023] The spatial graph convolution operation module and the spectral graph convolution attention module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

[0024] According to an achievable method in an embodiment of the present application, the preset spectral graph convolution feature fusion model includes a preset Chebyshev polynomial formula, a preset adjacency matrix weight formula, and a preset spatial graph convolution formula; according to the feature sample data of the scene image and the preset spectral graph convolution feature fusion model, a spectral graph convolution attention module of the scene image is obtained, including:

[0025] Obtaining local spectrogram convolution sample data of the scene image according to the feature sample data of the scene image and a preset Chebyshev polynomial formula;

[0026] The edge weight values ​​between local graph nodes are obtained according to the local spectral graph convolution sample data and the weight formula of the preset adjacency matrix;

[0027] The spectral graph convolution attention module of the scene image is obtained according to the edge weight values ​​between local graph nodes and the preset spatial graph convolution formula.

[0028] According to an achievable method in an embodiment of the present application, a scene image parsing module is obtained according to feature sample data of a scene image and a preset graph model, and a scene image parsing model architecture is constructed based on the scene image parsing module, further comprising:

[0029] According to the feature sample data of the scene image and the preset multi-domain adaptive graph pooling model, an adaptive graph pooling module of the scene image is obtained;

[0030] The spatial graph convolution operation module, the spectral graph convolution attention module and the adaptive graph pooling module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

[0031] According to an achievable method in an embodiment of the present application, the preset multi-domain adaptive graph pooling model includes a preset multi-domain adaptive formula and a preset graph pooling formula; according to the feature sample data of the scene image and the preset multi-domain adaptive graph pooling model, an adaptive graph pooling module of the scene image is obtained, including:

[0032] According to the feature sample data of the scene image and the preset multi-domain adaptive formula, an adaptive module of the scene image is obtained;

[0033] According to the adaptive module of the scene image and the preset image pooling formula, an adaptive image pooling module of the scene image is obtained.

[0034] According to an achievable method in an embodiment of the present application, a scene image parsing model architecture is trained to obtain a scene image parsing model, including:

[0035] Divide the image label sample data into training set data, validation set data and test set data;

[0036] Select the loss function and model training algorithm for model training and set the corresponding training configuration parameters;

[0037] The training set data is used to train the scene image parsing model architecture, the validation set data is used to verify the performance of the scene image parsing model architecture, and the test set data is used to test the performance of the scene image parsing model architecture to obtain the scene image parsing model.

[0038] In a second aspect, a scene image parsing system is provided, the system comprising:

[0039] The acquisition and annotation module is used to acquire multiple scene images as image sample data, label the image sample data, and obtain image label sample data;

[0040] A granularity division module is used to divide the image label sample data into grids according to a custom division granularity to obtain grid block sample data;

[0041] A first calculation module is used to establish a spatial hierarchical graph structure based on the grid block sample data, and extract feature vector data and assign the feature vector data to the spatial hierarchical graph structure to obtain feature sample data of the scene image;

[0042] A second calculation module is used to obtain a scene image parsing module according to the feature sample data of the scene image and the preset graph model, and to construct a scene image parsing model architecture based on the scene image parsing module;

[0043] The model training module is used to perform model training on the scene image parsing model architecture based on image label sample data to obtain a scene image parsing model.

[0044] In a third aspect, a computer device is provided, comprising:

[0045] at least one processor; and

[0046] a memory communicatively connected to at least one processor; wherein,

[0047] The memory stores computer instructions that can be executed by at least one processor, and the computer instructions are executed by at least one processor to enable the at least one processor to execute the method involved in the first aspect above.

[0048] In a fourth aspect, a computer-readable storage medium is provided, on which computer instructions are stored, characterized in that the computer instructions are used to enable a computer to execute the method involved in the above-mentioned first aspect.

[0049] According to the technical content provided by the embodiment of the present application, first, multiple scene images are collected as image sample data, and the image sample data are labeled to obtain image label sample data; the image label sample data is grid-divided according to the custom division granularity to obtain grid block sample data; then, a spatial hierarchical graph structure is established based on the grid block sample data, and feature vector data is extracted and assigned to the spatial hierarchical graph structure to obtain feature sample data of the scene image; a scene image parsing module is obtained based on the feature sample data of the scene image and a preset graph model, and a scene image parsing model architecture is constructed based on the scene image parsing module; finally, the scene image parsing model architecture is model-trained based on the image label sample data to obtain a scene image parsing model. The scene image parsing model obtained by the above operations can improve the small-scale target feature learning ability of the scene image parsing model and the accuracy of multi-scale parsing of the scene image parsing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A diagram of an application environment of a scene image parsing method in one embodiment;

[0051] Figure 2 A schematic diagram of a flow chart of a scene image parsing method in an embodiment;

[0052] Figure 3 A schematic diagram of a spatial graph structure in a scene image parsing method in one embodiment;

[0053] Figure 4 A schematic diagram of a process of establishing a spatial hierarchical graph structure in a scene image parsing method in one embodiment;

[0054] Figure 5 A schematic diagram of a spatial hierarchical graph structure in a scene image parsing method in one embodiment;

[0055] Figure 6 A schematic diagram of a process of extracting a feature vector and assigning the feature vector to a spatial hierarchical graph structure to obtain feature sample data of a scene image in a scene image analysis method in an embodiment;

[0056] Figure 7 A schematic diagram of a scene image parsing model architecture in a scene image parsing method in an embodiment;

[0057] Figure 8 is a structural block diagram of a scene image parsing system in one embodiment;

[0058] Fig. 9 FIG. 4 is a schematic structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] For ease of understanding, the system to which this application is applicable is first described. The scene image parsing method provided in this application can be applied to Figure 1 The system architecture shown in FIG. 1 includes: terminal device 102-network-server 104, and the terminal 102 communicates with the server 104 through the network. The terminal 102 can be, but is not limited to, various hardware devices such as personal computers, laptops, and tablet computers, and the server 104 can be implemented as an independent server or a server cluster composed of multiple servers.

[0061] Figure 2 A flowchart of a scene image parsing method provided in an embodiment of the present application, the method can be performed as follows Figure 1 The server 104 in the system architecture shown is executed. Figure 2 As shown, the method may include the following steps:

[0062] Step S201: collecting multiple scene images as image sample data, labeling the image sample data, and obtaining image label sample data;

[0063] Among them, scene images include images collected by cameras in natural scenes and artificial scenes, and artificial scenes are divided into indoor scenes and outdoor scenes, but no matter which scene it is, it belongs to the field of scene images.

[0064] Here, a large number of scene images are collected as image sample data. Since scene images contain many things, and things include people, animals or plants, man-made or natural objects, places or environments, etc., the image sample data is labeled, that is, a semantic category name can be defined for each thing contained in the scene image, and each semantic category name corresponds to an object category label; and since each scene image contains multiple pixels, a pixel category label is correspondingly labeled for each pixel according to the object to which each pixel belongs, thereby obtaining image label sample data.

[0065] In one feasible method, for example, the image label sample data is an image of an indoor scene of an office. Semantic category names are defined according to the objects contained in the indoor scene image, such as screen, printer, cup, window, etc., and the corresponding object category labels are defined as integers represented by 0, 1, 2, 3 in sequence; the pixel category label in the pixel area that is all screen is also 0, and so on. The pixel category labels of the indoor scene image are also 0, 1, 2, 3 and correspond one-to-one with the object category labels.

[0066] Step S203: grid-dividing the image label sample data according to the user-defined division granularity to obtain grid block sample data.

[0067] The custom partitioning granularity includes multiple partitioning granularities from coarse to fine.

[0068] Here, the image label sample data is divided into multiple granularities from coarse to fine, and each image label sample data is grid-divided into multiple granularities to obtain grid block sample data.

[0069] In a feasible way, the custom partitioning granularity includes three partitioning granularities from coarse to fine, corresponding to three spatial levels in turn, such as Figure 3 As shown in (a)(b)(c); wherein, the division granularity of spatial level-1, i.e., the grid block size, is 32×32 pixels, the division granularity of spatial level-2, i.e., the grid block size, is 16×16 pixels, and the division granularity of spatial level-3, i.e., the grid block size, is 8×8 pixels.

[0070] Step S205: establishing a spatial hierarchical graph structure based on the grid block sample data, extracting feature vector data, and assigning the feature vector data to the spatial hierarchical graph structure to obtain feature sample data of the scene image.

[0071] Here, since the feature sample data of the scene image contains two parts, one part is the spatial hierarchical graph structure, and the other part is the feature vector data, therefore, while establishing the spatial hierarchical graph structure based on the grid block sample data, it is also necessary to extract the feature vector data and assign the feature vector data to the spatial hierarchical graph structure, thereby obtaining the feature sample data of the scene image.

[0072] Step S207: obtaining a scene image parsing module according to the feature sample data of the scene image and the preset graph model, and constructing a scene image parsing model architecture based on the scene image parsing module.

[0073] Here, the feature sample data of the scene image is used as the input variable of the preset graph model, and then the output scene image parsing module can be obtained; the scene image parsing modules are then combined to obtain the graph convolutional network unit; finally, the stacking operation of the graph convolutional network unit is performed to complete the construction of the scene image parsing model architecture.

[0074] Step S209: Perform model training on the scene image parsing model architecture based on the image label sample data to obtain a scene image parsing model.

[0075] Here, the constructed scene image parsing model architecture is trained based on the image label sample data to obtain a scene image parsing model.

[0076] It can be seen that the embodiment of the present application collects multiple scene images as image sample data, labels the image sample data, and obtains image label sample data; the image label sample data is grid-divided according to the custom division granularity to obtain grid block sample data; then, a spatial hierarchical graph structure is established based on the grid block sample data, and feature vector data is extracted and assigned to the spatial hierarchical graph structure to obtain feature sample data of the scene image; a scene image parsing module is obtained according to the feature sample data of the scene image and the preset graph model, and a scene image parsing model architecture is constructed based on the scene image parsing module; finally, the scene image parsing model architecture is model-trained based on the image label sample data to obtain a scene image parsing model. This is to improve the learning ability of the small-scale target features of the scene image parsing model and the accuracy of the multi-scale parsing of the scene image parsing model.

[0077] The following is a detailed description of the individual steps in the above method flow. Figure 4 First, the "establishing a spatial hierarchical graph structure based on grid block sample data" in the above step 205 is described in detail in conjunction with the embodiment.

[0078] Step S301 : determining the grid category label of the grid block sample data according to the pixel category label of the image label sample data.

[0079] Here, for spatial level-1 / spatial level-2 / spatial level-3, their grid category labels can all be determined based on the pixel category labels in the image label sample data.

[0080] In an achievable manner, the specific basis for determining the grid category label of the grid block sample data according to the pixel category label of the image label sample data is as follows:

[0081] If the proportion of the number of pixels belonging to a certain pixel category label j in the grid block sample data is greater than or equal to the threshold α, then the grid category label corresponding to the grid block sample data is j; if the proportion of the number of pixels with different pixel category labels j in the grid block sample data is less than or equal to the threshold β, then the grid block sample data has no grid category label; if the proportion of the number of pixels with different pixel category labels j in the grid block sample data is less than the threshold α but greater than the threshold β, then the grid category label of the grid block sample data is the pixel category label j with the largest proportion of pixels. max Among them, the threshold α ranges from 0.6 to 0.8, and the threshold β ranges from 0.2 to 0.4.

[0082] For ease of understanding, here is an example. For example, at this time, there are three pixel category labels for a single grid block sample data, namely 1, 2 and 3; when the pixel number ratio of these three pixel category labels is less than or equal to the threshold β, the grid block sample data has no grid category label; when the pixel number ratio of the pixel category label 2 is greater than or equal to α, the grid category label of the grid block sample data is 2; when the pixel number ratio of these three pixel category labels is less than the threshold α but greater than the threshold β, the pixel category label with the largest pixel number ratio is the grid category label of the grid block sample data.

[0083] Step S303, establishing spatial edges according to the four-neighborhood relationship of the grid block sample data to obtain a spatial graph structure.

[0084] Among them, a graph node represents a grid block sample data, that is, each graph node contains all the information of the corresponding grid block sample data, and the grid category label of the grid block sample data is the node category label of the graph node.

[0085] Specifically, the spatial edge is established according to the four-neighborhood relationship of the grid block sample data to obtain the spatial graph structure. The spatial graph structure contains three disconnected subgraphs, such as Figure 3 (a) is shown; the screen above it corresponds to a subgraph consisting of 4 graph nodes, the screen on the lower left corresponds to a subgraph consisting of 2 graph nodes, and the printer on the lower right corresponds to a subgraph consisting of 1 graph node; and here for each graph node, the other graph nodes directly connected based on the four-neighborhood relationship form the first-level neighborhood, the other graph nodes indirectly connected through the graph nodes of the first-level neighborhood form the second-level neighborhood, and so on. Of course, here it can also be determined based on the length of the shortest path between the graph nodes. When the shortest path length is 1, it is the first-level neighborhood, when the length is 2, it is the second-level neighborhood, and so on.

[0086] Step S305 , connecting graph nodes of the same grid category label in adjacent spatial graph structures at the user-defined division granularity, establishing spatial hierarchical edges to obtain a spatial hierarchical graph structure.

[0087] Here, spatial level-1, spatial level-2, and spatial level-3 are superimposed, that is, the graph nodes with the same grid category labels in each adjacent spatial graph structure are connected, and spatial level edges are established to obtain a spatial level graph structure. Figure 5 As shown in the figure, the edge connecting the graph nodes of the same grid category label belonging to two adjacent spatial graph structures is called a spatial hierarchical edge. The spatial hierarchical graph structure here can be formally expressed as the following formula:

[0088]

[0089] in, is the set of graph nodes, ε S is the set of spatial edges, ε SH is the set of spatial hierarchical edges, Λ S is the adjacency matrix of the spatial edge, Λ SH is the adjacency matrix of the spatial level edges.

[0090] Reference Figure 6 , the following is a detailed description of "extracting feature vector data and assigning the feature vector data to the spatial hierarchical graph structure to obtain feature sample data of the scene image" in the above step 205 in conjunction with the embodiment.

[0091] Step S401, using a feature extraction algorithm to extract features from grid block sample data to obtain feature vector data.

[0092] Here, there are many types of feature extraction algorithms, such as HOG, SIFT, GIST, and LBP. Since each feature extraction algorithm has different characteristics, the corresponding parameter settings are also different. The specific feature extraction algorithm can be formally expressed as the following formula:

[0093] x=f(x,δ)

[0094] Among them, f represents the feature extraction algorithm function, δ represents the configuration parameters corresponding to the feature extraction algorithm, x represents the pixel value of the input grid block sample data, and x represents the feature vector data output after the feature extraction algorithm.

[0095] Step S403, assigning the feature vector data to the corresponding graph nodes in the spatial hierarchical graph structure to obtain feature sample data of the scene image.

[0096] Specifically, the feature vector data is assigned to the corresponding graph node in the spatial hierarchical graph structure, that is, the obtained feature sample data of the scene image includes the spatial hierarchical graph structure and the feature vector data, which can be formally expressed as the following formula:

[0097]

[0098] Among them, x n,l is a graph node v n,l characteristic data; for dimensional real number, and is the number of feature channels; the number of spatial levels is The number of graph nodes at layer l is

[0099] Reference Figure 7Next, the above step 207, namely, "obtaining a scene image parsing module according to the feature sample data of the scene image and the preset graph model, and constructing a scene image parsing model architecture based on the scene image parsing module" is described in detail in conjunction with an embodiment.

[0100] In some embodiments, a spatial graph convolution operation module of the scene image is obtained based on feature sample data of the scene image and a preset spatial graph convolution formula; the spatial graph convolution operation modules are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

[0101] In one achievable manner, the preset spatial graph convolution model may be a preset spatial graph convolution formula based on a K-level neighborhood, which is specifically expressed as follows:

[0102]

[0103] Among them, x v is the characteristic sample data of graph node v, is the feature sample weight of graph node c, θ represents the edge connection weight of the neighborhood of the previous layer node, K is the neighborhood level, is the set of K-level neighboring nodes of graph node v; (t) represents the (t)th layer graph convolution operation, and there is

[0104] It should be noted that To solve the problem of θ and θ, before model training, the parameters The values ​​of and θ are randomly initialized. Here, the error back propagation algorithm is generally used to adjust the parameters and θ are updated, which can be formally expressed as follows:

[0105]

[0106]

[0107] Among them, ← represents parameter assignment update, r w and r θ is the model parameter learning rate, is the gradient, and its value range is 10 -5 ~10 -1 It should be noted here that since the error back propagation algorithm is well known to those skilled in the art in model training, it will not be described in detail here, but only briefly represented.

[0108] After obtaining the spatial graph convolution operation module of the scene image through the above calculation, the spatial graph convolution operation modules are combined to obtain the graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit. The scene image parsing model architecture constructed by the spatial graph convolution operation module based on the K-level neighborhood is realized, so as to improve the learning ability of the global target features of the scene image parsing model and achieve the accuracy of multi-scale scene image parsing.

[0109] In some embodiments, a spectral convolution attention module of the scene image is obtained based on feature sample data of the scene image and a preset spectral convolution feature fusion model; the spatial graph convolution operation module and the spectral convolution attention module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

[0110] In one achievable manner, the preset spectral graph convolution feature fusion model includes a preset Chebyshev polynomial formula, a preset adjacency matrix weight formula, and a preset spatial graph convolution formula.

[0111] Here, according to the connection relationship between graph nodes, graph nodes with less than 4 level 1 neighborhood connections are selected, and their multi-level neighborhood graph nodes are extracted to form independent local graph structures, and then a local spectral graph convolution model is established to realize the feature fusion mechanism. Among them, the local spectral graph convolution operation can use Chebyshev polynomials.

[0112] Specifically, the local spectrogram convolution sample data of the scene image is obtained according to the feature sample data of the scene image and the preset Chebyshev polynomial first-order approximation or second-order approximation formula, and the specific calculation is as follows:

[0113]

[0114]

[0115] Among them, x represents the characteristic sample data, * G represents the spectral graph convolution operation, g σ represents the convolution kernel, D represents the diagonal matrix of the local graph nodes, A represents the adjacency matrix of the local graph nodes, σ0, σ1 and σ2 are all polynomial parameters, and their values ​​range from 0 to 1.

[0116] The edge weight values ​​between local graph nodes are obtained according to the weight formula of the local spectral graph convolution sample data and the preset adjacency matrix. Here, the local spectral graph convolution sample data is used as self-supervisory information, and the edge weight e between local graph nodes v and u is uv The calculation formula is as follows:

[0117]

[0118] in, is a learnable parameter matrix, exp(·) is the exponential function, and ||·|| represents the vector norm.

[0119] The spectral graph convolution attention module of the scene image is obtained according to the edge weight value between the local graph nodes and the preset spatial graph convolution formula, that is, the self-supervised spectral graph convolution attention module of the scene image is obtained by multiplying the edge weight value between the local graph nodes and the preset spatial graph convolution formula. The specific formula is expressed as:

[0120]

[0121] The spectral graph convolution attention module of the scene image is obtained through the above calculation, and then the spectral graph convolution attention module of the scene image and the spatial graph convolution operation module are combined to obtain the graph convolution network unit, and then the scene image parsing model architecture is constructed based on the graph convolution network unit. In this way, the local feature learning ability of the scene image parsing model is improved, and the accuracy of multi-scale scene image parsing is achieved.

[0122] In some embodiments, an adaptive graph pooling module of the scene image is obtained based on feature sample data of the scene image and a preset multi-domain adaptive graph pooling model; the spatial graph convolution operation module, the spectral graph convolution attention module and the adaptive graph pooling module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

[0123] In an achievable manner, the preset multi-domain adaptive graph pooling model includes a preset multi-domain adaptive formula and a preset graph pooling formula; an adaptive module of the scene image is obtained according to the feature sample data of the scene image and the preset multi-domain adaptive graph pooling model; that is, the feature sample data of the scene image is applied to the preset multi-domain adaptive formula, and the feature gain parameter μ is set. gain , characteristic nonlinear parameter μ gamma And the feature bias parameter μ offset , we get the adaptive module of the scene image, which can be formalized as follows:

[0124]

[0125] Among them, x ref Represents the feature sample data of the original scene image, x tar Represents feature sample data of the transformed scene image. is the normalization parameter, The value of can be determined according to the feature sample data of the input scene image. When the feature sample data of the input scene image is the pixel value of the scene image data sample, d is the pixel width; when the feature sample data of the input scene image is the feature sample data obtained by the feature extraction algorithm, It can be determined according to the value of the number of feature channels.

[0126] Here, the parameter μ gain , μ gamma and μ offset It can also be formally expressed through the error back propagation algorithm as follows:

[0127]

[0128] μ=[μ gain ,μ gamma ,μ offset ]

[0129] Among them, r μ is the parameter learning rate, and its value range is 10 -5 ~10 -1 It should be noted that since the error back propagation algorithm is well known to those skilled in the art in model training, it will not be described in detail here, and only a simple representation will be given.

[0130] According to the adaptive module of the scene image and the preset image pooling formula, the adaptive image pooling module of the scene image is obtained. The specific preset image pooling formula can adopt the maximum value or average value pooling calculation, and its formula is expressed as follows:

[0131]

[0132]

[0133] Among them, pool max (·) represents the maximum value pooling operation, pool avg (·) represents the mean map pooling operation.

[0134] Since the adaptive graph pooling module consists of two parts, one is to adaptively learn and pool the feature sample data of the graph nodes, and the other is to pool the connection structure of the graph nodes. Through the above operation, the adaptive learning and pooling of the feature sample data of the graph nodes is realized.

[0135] The main operation of pooling the connection structure of graph nodes is to reduce graph nodes or edges to obtain a new graph connection structure. It adopts the subgraph clustering formula based on graph neural network, which can be formally expressed as follows:

[0136] G′=pool G (G)

[0137] Among them, pool G (·) represents the subgraph clustering pooling formula, G represents the original graph connection structure, and G′ represents the graph connection structure after pooling.

[0138] After obtaining the adaptive graph pooling module of the scene image through the above calculation, the spatial graph convolution operation module, the spectral graph convolution attention module and the adaptive graph pooling module are combined to obtain the graph convolution network unit, and the scene image parsing model architecture is constructed based on the graph convolution network unit. This can improve the inconsistency problem within the class caused by the differences of the scene images themselves and improve the learning ability of the local features of the scene image parsing model, so as to achieve the accuracy of multi-scale scene image parsing.

[0139] In some embodiments, the image label sample data is divided into training set data, validation set data and test set data; the loss function and model training algorithm of the model training are selected, and the corresponding training configuration parameters are set; the training set data is used to train the scene image parsing model architecture, the validation set data is used to verify the performance of the scene image parsing model architecture, and the test set data is used to test the performance of the scene image parsing model architecture to obtain a scene image parsing model.

[0140] In one feasible method, the image label sample data is divided into three data sets, namely, training set data, validation set data and test set data. It should be emphasized here that when the image label sample data is divided into three data sets, the original distribution of the image sample data should be kept unchanged as much as possible.

[0141] Select the loss function and model training algorithm for model training and set the corresponding training configuration parameters.

[0142] In one achievable approach, the loss function can be a cross entropy function, which is specifically expressed as follows:

[0143]

[0144] Among them, y * is the true value label of the graph node, y is the predicted output of model training, and η is the balance weight, which ranges from 0 to 1.

[0145] The model training algorithm can adopt the error back propagation algorithm, use the cross entropy function as the objective function, and use the stochastic gradient descent method to achieve the optimal setting of model training parameters. The specific form is as follows:

[0146]

[0147] Among them, ω is the generalized model parameter, specifically referring to the above-mentioned w, θ and μ, and the generalized training configuration parameter r ω is the model parameter learning rate, specifically the r mentioned above w 、r θ and r μ , its value range is 10-5 ~10 -1 The training configuration parameter T here is the maximum number of iterations of the model training algorithm, which is 500,000 times. It should also be noted that since the error back propagation algorithm is well known to those skilled in the art in model training, it will not be described in detail here, but only briefly indicated.

[0148] The training set data is used to train the scene image parsing model architecture, the validation set data is used to verify the performance of the scene image parsing model architecture, and the test set data is used to test the performance of the scene image parsing model architecture, thereby obtaining the scene image parsing model.

[0149] In one feasible method, the model training algorithm is iterated 100 times and the validation set is used to verify the model performance once. If the difference in model performance after five consecutive verifications is less than 0.1, the model verified for the fifth time is used as a candidate scene parsing model. The test set data is then used to perform a performance test on the candidate scene parsing models, and the candidate scene parsing model with the best test performance is used as the final scene image parsing model.

[0150] After the scene image parsing model is obtained, a new scene image is collected and directly input into the scene image parsing model to obtain the semantic parsing result of the scene image.

[0151] It should be understood that although Figure 2 , Figure 4 as well as Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in the application, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , Figure 4 as well as Figure 6 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0152] Figure 8 A schematic diagram of a scene image analysis system provided in an embodiment of the present application, which can be arranged in Figure 1 In the system architecture shown, the following are used to execute Figure 2 , Figure 4 , Figure 6 The method flow shown in Figure 8As shown, the system may include: a collection and annotation module 501, a granularity division module 503, a first calculation module 505, a second calculation module 507, and a model training module 509. The main functions of each component module are as follows:

[0153] The collection and labeling module 501 is used to collect multiple scene images as image sample data, label the image sample data, and obtain image label sample data;

[0154] The granularity division module 503 is used to divide the image label sample data into grids according to the custom division granularity to obtain grid block sample data;

[0155] The first calculation module 505 is used to establish a spatial hierarchical graph structure based on the grid block sample data, and extract feature vector data and assign the feature vector data to the spatial hierarchical graph structure to obtain feature sample data of the scene image;

[0156] A second calculation module 507, configured to obtain a scene image parsing module according to the feature sample data of the scene image and a preset graph model, and to construct a scene image parsing model architecture based on the scene image parsing module;

[0157] The model training module 509 is used to perform model training on the scene image parsing model architecture based on the image label sample data to obtain the scene image parsing model.

[0158] In some embodiments, the first calculation module 505 is further configured to:

[0159] Determine the grid category label of the grid block sample data according to the pixel category label of the image label sample data;

[0160] According to the four-neighborhood relationship of the grid block sample data, a spatial edge is established to obtain a spatial graph structure; wherein a graph node represents a grid block sample data;

[0161] The graph nodes with the same grid category labels in each adjacent spatial graph structure under the custom partitioning granularity are connected, and spatial hierarchical edges are established to obtain a spatial hierarchical graph structure.

[0162] In some embodiments, the first calculation module 505 is further configured to:

[0163] The feature extraction algorithm is used to extract features from the grid block sample data to obtain feature vector data;

[0164] The feature vector data is assigned to the corresponding graph nodes in the spatial hierarchical graph structure to obtain the feature sample data of the scene image.

[0165] In some embodiments, the second calculation module 505 is further configured to:

[0166] According to the feature sample data of the scene image and the preset spatial graph convolution formula, a spatial graph convolution operation module of the scene image is obtained;

[0167] The spatial graph convolution operation modules are combined to obtain the graph convolution network unit, and the scene image parsing model architecture is constructed based on the graph convolution network unit.

[0168] In some embodiments, the second calculation module 505 is further configured to:

[0169] According to the feature sample data of the scene image and the preset spectral convolution feature fusion model, a spectral convolution attention module of the scene image is obtained;

[0170] The spatial graph convolution operation module and the spectral graph convolution attention module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

[0171] In some embodiments, the "preset spectral graph convolution feature fusion model includes a preset Chebyshev polynomial formula, a preset adjacency matrix weight formula, and a preset spatial graph convolution formula; according to the feature sample data of the scene image and the preset spectral graph convolution feature fusion model, a spectral graph convolution attention module of the scene image is obtained" in the second computing module 505 is also used to:

[0172] Obtaining local spectrogram convolution sample data of the scene image according to the feature sample data of the scene image and a preset Chebyshev polynomial formula;

[0173] The edge weight values ​​between local graph nodes are obtained according to the local spectral graph convolution sample data and the weight formula of the preset adjacency matrix;

[0174] The spectral graph convolution attention module of the scene image is obtained according to the edge weight values ​​between local graph nodes and the preset spatial graph convolution formula.

[0175] In some embodiments, the second calculation module 505 is further configured to:

[0176] According to the feature sample data of the scene image and the preset multi-domain adaptive graph pooling model, an adaptive graph pooling module of the scene image is obtained;

[0177] The spatial graph convolution operation module, the spectral graph convolution attention module and the adaptive graph pooling module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

[0178] In some embodiments, the "preset multi-domain adaptive graph pooling model includes a preset multi-domain adaptive formula and a preset graph pooling formula; according to the feature sample data of the scene image and the preset multi-domain adaptive graph pooling model, an adaptive graph pooling module of the scene image is obtained" in the second calculation module 505 is also used to:

[0179] According to the feature sample data of the scene image and the preset multi-domain adaptive formula, an adaptive module of the scene image is obtained;

[0180] According to the adaptive module of the scene image and the preset image pooling formula, an adaptive image pooling module of the scene image is obtained.

[0181] In some embodiments, the model training module 509 is further used to:

[0182] Divide the image label sample data into training set data, validation set data and test set data;

[0183] Select the loss function and model training algorithm for model training and set the corresponding training configuration parameters;

[0184] The training set data is used to train the scene image parsing model architecture, the validation set data is used to verify the performance of the scene image parsing model architecture, and the test set data is used to test the performance of the scene image parsing model architecture to obtain the scene image parsing model.

[0185] The same or similar parts between the above embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0186] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described in this article within the scope permitted by applicable laws and regulations, provided that the applicable laws and regulations of the country are met (for example, the user's explicit consent, effective notification to the user, and the user's explicit authorization, etc.).

[0187] According to an embodiment of the present application, the present application also provides a computer device and a computer-readable storage medium.

[0188] like Fig. 9 , is a block diagram of a computer device according to an embodiment of the present application. The computer device is intended to represent various forms of digital computers or mobile devices. The digital computer may include a desktop computer, a portable computer, a workbench, a personal digital assistant, a server, a mainframe computer, and other suitable computers. The mobile device may include a tablet computer, a smart phone, a wearable device, etc.

[0189] like Fig. 9As shown, the device 900 includes a computing unit 901, a ROM 902, a RAM 903, a bus 904, and an input / output (I / O) interface 905, and the computing unit 901, ROM 902, and RAM 903 are connected to each other through the bus 904. The input / output (I / O) interface 905 is also connected to the bus 904.

[0190] The computing unit 901 can perform various processes in the method embodiment of the present application according to the computer instructions stored in the read-only memory (ROM) 902 or the computer instructions loaded from the storage unit 908 to the random access memory (RAM) 903. The computing unit 901 can be various general and / or special processing components with processing and computing capabilities. The computing unit 901 may include, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. In some embodiments, the method provided in the embodiment of the present application can be implemented as a computer software program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 908.

[0191] The RAM 903 may also store various programs and data required for the operation of the device 900. Part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909.

[0192] The input unit 906, output unit 907, storage unit 908 and communication unit 909 in the device 900 can be connected to the I / O interface 905. The input unit 906 can be, for example, a keyboard, a mouse, a touch screen, a microphone, etc. The output unit 907 can be, for example, a display, a speaker, an indicator light, etc. The device 900 can exchange information, data, etc. with other devices through the communication unit 909.

[0193] It should be noted that the device may also include other components necessary for normal operation, or may only include components necessary for implementing the solution of the present application, rather than all the components shown in the figure.

[0194] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof.

[0195] The computer instructions for implementing the method of the present application can be written in any combination of one or more programming languages. These computer instructions can be provided to the computing unit 901, so that when the computer instructions are executed by the computing unit 901 such as a processor, the steps involved in the method embodiment of the present application are executed.

[0196] The computer-readable storage medium provided in the present application may be a tangible medium that may contain or store computer instructions for executing the steps involved in the method embodiments of the present application. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, and other forms of storage media.

[0197] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A scene image analysis method, characterized in that: The method includes: Collecting multiple scene images as image sample data, labeling the image sample data, and obtaining image label sample data; Divide the image label sample data into grids according to a custom division granularity to obtain grid block sample data; Determine the grid category label of the grid block sample data according to the pixel category label of the image label sample data; Establishing spatial edges according to the four-neighborhood relationship of the grid block sample data to obtain a spatial graph structure; wherein one graph node represents one grid block sample data; Connect the graph nodes with the same grid category labels in each adjacent spatial graph structure under the custom partitioning granularity, establish spatial hierarchical edges to obtain a spatial hierarchical graph structure; Using a feature extraction algorithm to extract features from the grid block sample data to obtain feature vector data; Assigning the feature vector data to corresponding graph nodes in the spatial hierarchical graph structure to obtain feature sample data of the scene image; Obtaining a scene image parsing module according to the feature sample data of the scene image and a preset graph model, and constructing a scene image parsing model architecture based on the scene image parsing module; Model training is performed on the scene image parsing model architecture based on the image label sample data to obtain a scene image parsing model.

2. The method according to claim 1, characterized in that The step of obtaining a scene image parsing module according to the feature sample data of the scene image and a preset graph model, and constructing a scene image parsing model architecture based on the scene image parsing module includes: Obtaining a spatial graph convolution operation module for the scene image according to the feature sample data of the scene image and a preset spatial graph convolution formula; The spatial graph convolution operation modules are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

3. The method according to claim 2, characterized in that The step of obtaining a scene image parsing module according to the feature sample data of the scene image and a preset graph model, and constructing a scene image parsing model architecture based on the scene image parsing module, further includes: Obtaining a spectral convolution attention module of the scene image according to the feature sample data of the scene image and a preset spectral convolution feature fusion model; The spatial graph convolution operation module and the spectral graph convolution attention module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

4. The method according to claim 3, characterized in that The preset spectral graph convolution feature fusion model includes a preset Chebyshev polynomial formula, a preset adjacency matrix weight formula and a preset spatial graph convolution formula; the spectral graph convolution attention module of the scene image is obtained according to the feature sample data of the scene image and the preset spectral graph convolution feature fusion model, including: Obtaining local spectrogram convolution sample data of the scene image according to the feature sample data of the scene image and a preset Chebyshev polynomial formula; Obtain edge weight values ​​between local graph nodes according to the local spectral graph convolution sample data and a weight formula of a preset adjacency matrix; A spectral graph convolution attention module of the scene image is obtained according to the edge weight values ​​between the local graph nodes and the preset spatial graph convolution formula.

5. The method according to any one of claims 3 or 4, characterized in that: The step of obtaining a scene image parsing module according to the feature sample data of the scene image and a preset graph model, and constructing a scene image parsing model architecture based on the scene image parsing module, further includes: Obtaining an adaptive graph pooling module for the scene image according to the feature sample data of the scene image and a preset multi-domain adaptive graph pooling model; The spatial graph convolution operation module, the spectral graph convolution attention module and the adaptive graph pooling module are combined to obtain a graph convolution network unit, and a scene image parsing model architecture is constructed based on the graph convolution network unit.

6. The method according to claim 5, characterized in that The preset multi-domain adaptive graph pooling model includes a preset multi-domain adaptive formula and a preset graph pooling formula; The method of obtaining an adaptive graph pooling module for the scene image according to the feature sample data of the scene image and a preset multi-domain adaptive graph pooling model comprises: Obtaining an adaptive module of the scene image according to the feature sample data of the scene image and a preset multi-domain adaptive formula; According to the adaptive module of the scene image and the preset image pooling formula, an adaptive image pooling module of the scene image is obtained.

7. The method according to any one of claims 1 to 4, characterized in that: The performing model training on the scene image parsing model architecture to obtain a scene image parsing model includes: Dividing the image label sample data into training set data, validation set data and test set data; Select the loss function and model training algorithm for model training and set the corresponding training configuration parameters; The training set data is used to train the scene image parsing model architecture, the validation set data is used to verify the performance of the scene image parsing model architecture, and the test set data is used to test the performance of the scene image parsing model architecture to obtain the scene image parsing model.

8. A scene image analysis system, characterized in that: The system includes: An acquisition and annotation module is used to acquire multiple scene images as image sample data, label the image sample data, and obtain image label sample data; A granularity division module, used to divide the image label sample data into grids according to a custom division granularity to obtain grid block sample data; The first calculation module is used to determine the grid category label of the grid block sample data according to the pixel category label of the image label sample data; establish a spatial edge according to the four-neighborhood relationship of the grid block sample data to obtain a spatial graph structure; wherein one graph node represents one grid block sample data; connect the graph nodes of the same grid category label in each adjacent spatial graph structure under the custom division granularity, establish a spatial hierarchical edge to obtain a spatial hierarchical graph structure; use a feature extraction algorithm to extract features from the grid block sample data to obtain feature vector data; assign the feature vector data to the corresponding graph node in the spatial hierarchical graph structure to obtain feature sample data of the scene image; A second calculation module, configured to obtain a scene image parsing module according to the feature sample data of the scene image and a preset graph model, and to construct a scene image parsing model architecture based on the scene image parsing module; The model training module is used to perform model training on the scene image parsing model architecture based on the image label sample data to obtain a scene image parsing model.

Images