Image semantic segmentation method and device, computer device and storage medium

Abstract

The application relates to an image semantic segmentation method and device, computer equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a target image; acquiring a first vector matrix corresponding to the target image in a query dimension, a second vector matrix in a key dimension and a third vector matrix in a value dimension; performing n times of position exchange on numerical values in the target vector matrix to obtain n extended vector matrices; acquiring a self-attention vector matrix based on the first vector matrix, the second vector matrix and the n extended vector matrices; acquiring an output vector matrix based on the self-attention vector matrix and the third vector matrix; and acquiring a semantic segmentation result corresponding to the target image based on the output vector matrix. Through the above method, the space information contained in different feature dimensions between K and Q can be fully mined by using the extended vector matrix in the process of acquiring the semantic segmentation result of the target image, and the precision of image semantic segmentation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image semantic segmentation method and device, a computer device and a storage medium. BACKGROUND

[0002] Image semantic segmentation is one of the three basic tasks in the field of machine vision, and is widely used in various industries. For example, autonomous driving, human-computer interaction, augmented reality, etc.

[0003] In related technologies, a segmentation framework based on a transformer is provided to obtain global information of a processed image.

[0004] However, the segmentation framework based on the transformer still has the problem that the pixel spatial position does not correspond, resulting in poor performance of image semantic segmentation. SUMMARY

[0005] Embodiments of the present application provide an image semantic segmentation method, device, computer device and storage medium, which can improve the accuracy of image semantic segmentation. The technical solution is as follows:

[0006] On the one hand, an image semantic segmentation method is provided, which comprises:

[0007] obtaining a target image;

[0008] obtaining a first vector matrix corresponding to the target image in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension;

[0009] performing n times of position exchange on the numerical values in the target vector matrix to obtain n times of extended vector matrices corresponding to the n times of position exchange respectively; the target vector matrix is at least one of the first vector matrix and the second vector matrix; n is a positive integer;

[0010] obtaining a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the n times of extended vector matrices corresponding to the n times of position exchange respectively;

[0011] obtaining an output vector matrix based on the self-attention vector matrix and the third vector matrix;

[0012] obtaining a semantic segmentation result corresponding to the target image based on the output vector matrix.

[0013] On the other hand, an image semantic segmentation method is provided, which comprises:

[0014] obtaining a target image;

[0015] The first vector matrix corresponding to the target image in a query dimension, the second vector matrix in a key dimension, and the third vector matrix in a value dimension are obtained through a cyclic transformer in an image semantic segmentation model; n times of position exchange are performed on values in a target vector matrix to obtain extended vector matrices corresponding to the n times of position exchange respectively; a self-attention vector matrix of the target image is obtained based on the first vector matrix, the second vector matrix, and the extended vector matrices corresponding to the n times of position exchange respectively; an output vector matrix is obtained based on the attention vector matrix and the third vector matrix; the target vector matrix is at least one of the first vector matrix and the second vector matrix;

[0016] A semantic segmentation result corresponding to the target image is obtained based on the output vector matrix through a decoder in the image semantic segmentation model.

[0017] In another aspect, an image semantic segmentation device is provided, and the device comprises:

[0018] A target image obtaining module is configured to obtain a target image.

[0019] A first obtaining module is configured to obtain a first vector matrix corresponding to the target image in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension.

[0020] A position exchange module is configured to perform n times of position exchange on values in a target vector matrix to obtain extended vector matrices corresponding to the n times of position exchange respectively; the target vector matrix is at least one of the first vector matrix and the second vector matrix.

[0021] A second obtaining module is configured to obtain a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the extended vector matrices corresponding to the n times of position exchange respectively.

[0022] A third obtaining module is configured to obtain an output vector matrix based on the attention vector matrix and the third vector matrix.

[0023] A semantic segmentation result obtaining module is configured to obtain a semantic segmentation result corresponding to the target image based on the output vector matrix.

[0024] In a possible implementation, the first vector matrix, the second vector matrix, and the third vector matrix respectively contain feature vectors of respective image blocks in the target image.

[0025] The position exchange module is configured to perform n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices corresponding to the n times of position exchange respectively.

[0026] In a possible implementation, the position exchange module is configured to perform n times of cyclic position exchange on values in the target vector matrix in units of the feature vectors according to the specified step length to obtain the extended vector matrices corresponding to the n times of cyclic position exchange respectively.

[0027] In a possible implementation, the position exchange module includes:

[0028] The queue obtaining submodule is configured to obtain a feature vector queue composed of the target vector matrix.

[0029] The position exchange submodule is configured to, in response to one time of cyclic position exchange, move the feature vectors with the specified step length at the end of the feature vector queue to the beginning of the feature vector queue, and sequentially move other feature vectors in the feature vector queue.

[0030] In a possible implementation, the position exchange submodule is further configured to, in response to the number of times of cyclic position exchange reaching a threshold number of times, stop the cyclic position exchange, where the threshold number of times is determined based on the number of the feature vectors in the target vector matrix and the specified step length.

[0031] In a possible implementation, the position exchange module is configured to randomly perform n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices corresponding to the n times of position exchange respectively.

[0032] In a possible implementation, the second obtaining module includes:

[0033] The first obtaining submodule is configured to obtain n dot product results of the first vector matrix and the extended vector matrices corresponding to the n times of position exchange respectively.

[0034] The second obtaining submodule is configured to obtain an average value of the n dot product results.

[0035] The third obtaining submodule is configured to obtain the self-attention vector matrix of the target image based on the average value.

[0036] In a possible implementation, the apparatus further includes:

[0037] The serialization processing module is configured to serialize the target image before obtaining a first vector matrix corresponding to the target image in a query dimension, a second vector matrix corresponding to the target image in a key dimension, and a third vector matrix corresponding to the target image in a value dimension.

[0038] The first obtaining module is configured to obtain the first vector matrix, the second vector matrix, and the third vector matrix of the target image based on the one-dimensional sequence.

[0039] In a possible implementation, the serialization processing module includes:

[0040] The image division submodule is configured to divide the target image into N image blocks, where N is greater than or equal to 2 and is a positive integer.

[0041] The fourth obtaining submodule is configured to obtain a vector matrix composed of image block vectors corresponding to the N image blocks.

[0042] The linear transformation submodule is configured to perform linear transformation on the vector matrix composed of the N image block vectors to obtain the one-dimensional sequence corresponding to the target image.

[0043] In a possible implementation, the first obtaining module is configured to obtain x different vector matrix groups based on x different weight matrices, where each of the vector matrix groups includes a corresponding first vector submatrix, a second vector submatrix, and a third vector submatrix; the first vector matrix includes x first vector submatrices, the second vector matrix includes x second vector submatrices, and the third vector matrix includes x third vector submatrices; and x is a positive integer.

[0044] The position exchange module is configured to perform n times of position exchange on values in a target vector submatrix corresponding to each of the x vector matrix groups to obtain n times of extended vector submatrices corresponding to the n times of position exchange; the target vector submatrix is at least one of the first vector submatrix and the second vector submatrix; and the extended vector matrix includes x extended vector submatrices corresponding to the x vector matrix groups.

[0045] The second obtaining module is configured to obtain self-attention vector submatrices of x target images corresponding to the x vector matrix groups based on the first vector submatrices, the second vector submatrices, and the n times of extended vector submatrices corresponding to the n times of position exchange of the x vector matrix groups; and the self-attention vector matrix includes x self-attention vector submatrices corresponding to the x vector matrix groups.

[0046] The third acquisition module is configured to acquire the output vector matrix based on the x self-attention vector sub-matrices and the third vector matrix.

[0047] In a possible implementation, the third acquisition module comprises:

[0048] A product result acquisition submodule is configured to acquire a product result obtained by multiplying the self-attention vector sub-matrix of the target image corresponding to each of the vector matrix groups and the third vector matrix;

[0049] A concatenation submodule is configured to concatenate the x product results to obtain a concatenation result.

[0050] A projection submodule is configured to project the concatenation result to obtain the output vector matrix.

[0051] In another aspect, an image semantic segmentation device is provided, and the device comprises:

[0052] A target image acquisition module is configured to acquire a target image.

[0053] An output vector matrix acquisition module is configured to acquire, by a recurrent Transformer in an image semantic segmentation model, a first vector matrix in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension corresponding to the target image; perform n times of position exchange on values in a target vector matrix to obtain n extended vector matrices corresponding to the n times of position exchange respectively; acquire self-attention vector matrices of the target image based on the first vector matrix, the second vector matrix, and the n extended vector matrices corresponding to the n times of position exchange respectively; and acquire an output vector matrix based on the attention vector matrices and the third vector matrix; the target vector matrix is at least one of the first vector matrix and the second vector matrix.

[0054] A semantic segmentation result acquisition module is configured to acquire, by a decoder in the image semantic segmentation model, a semantic segmentation result corresponding to the target image based on the output vector matrix.

[0055] In another aspect, a computer device is provided, and the computer device comprises a processor and a memory, the memory stores at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-mentioned image semantic segmentation method.

[0056] In another aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores at least one computer program. The computer program is loaded and executed by a processor to implement the image semantic segmentation method described above.

[0057] In another aspect, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image semantic segmentation method provided in various optional implementations described above.

[0058] The technical solutions provided in the present application can include the following beneficial effects:

[0059] After obtaining the first vector matrix, the second vector matrix, and the third vector matrix corresponding to the target image in the Q, K, and V dimensions, respectively, the target vector matrix in the first vector matrix and the second vector matrix is exchanged n times to obtain n extended vector matrices. Based on the n extended vector matrices, the first vector matrix, and the second vector matrix, a self-attention vector matrix of the target image is obtained. Based on the output vector matrix obtained from the self-attention vector matrix and the third vector matrix, a semantic segmentation result of the target image is obtained. In the process of obtaining the semantic segmentation result of the target image, the extended vector matrix can be used to fully exploit the spatial information contained in different feature dimensions between K and Q, thereby improving the accuracy and performance of image semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0061] Figure 1 A schematic diagram of a system used by an image semantic segmentation method provided by an exemplary embodiment of the present application is shown;

[0062] Figure 2 A flowchart of an image semantic segmentation method shown by an exemplary embodiment of the present application is shown;

[0063] Figure 3 is a framework diagram of an image semantic segmentation model generation and image semantic segmentation according to an exemplary embodiment of the present application;

[0064] Figure 4 is a flowchart of a training method of an image semantic segmentation model according to an exemplary embodiment of the present application;

[0065] Figure 5A schematic diagram of an image semantic segmentation model according to an example embodiment of the present application is shown.

[0066] Figure 6 A flowchart of an image semantic segmentation method according to an example embodiment of the present application is shown.

[0067] Figure 7 A schematic diagram of a cyclic position exchange according to an example embodiment of the present application is shown.

[0068] Figure 8 A schematic diagram of a decoder according to an example embodiment of the present application is shown.

[0069] Figure 9 A block diagram of an image semantic segmentation apparatus according to an example embodiment of the present application is shown.

[0070] Figure 10 A block diagram of an image semantic segmentation apparatus according to an example embodiment of the present application is shown.

[0071] Figure 11 A structural block diagram of a computer device according to an example embodiment of the present application is shown.

[0072] Figure 12 A structural block diagram of a computer device according to an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0073] The example embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to accompanying drawings, unless otherwise specified, the same numbers in different drawings represent the same or similar elements. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0074] It should be understood that "multiple" referred to herein means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0075] The embodiments of the present application provide an image semantic segmentation method, which can improve the corresponding effect of the pixel spatial position and improve the performance of image semantic segmentation. In order to facilitate understanding, the following explains several terms related to the present application.

[0076] 1) Artificial Intelligence (AI)

[0077] Artificial intelligence is the theory, method, technology and application system of using digital computer or machine controlled by digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0078] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc. The display device comprising an image acquisition component shown in the present application mainly relates to computer vision technology and machine learning / deep learning, autonomous driving, intelligent transportation, etc.

[0079] 2) Machine Learning (ML)

[0080] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0081] 3) Computer Vision (CV)

[0082] Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics so that the computer processing becomes images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology generally includes image semantic segmentation, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0083] 4) Self Attention Mechanism

[0084] The role of the self-attention mechanism is to better learn the dependency relationship between global features. Self-attention directly calculates the relationship between any two nodes in the graph structure to obtain the global geometric features of the graph structure.

[0085] Self-attention uses the attention mechanism and is calculated in three stages: (1) different functions and computing mechanisms are introduced to calculate the similarity or correlation between Query and a certain Key. The most common methods include: calculating the vector dot product of the two, calculating the vector similarity of the two, or introducing an additional neural network to evaluate; (2) a similar softmax calculation method is introduced to convert the scores of the first stage into numerical values. On the one hand, it can normalize the original calculation score to a probability distribution with a sum of all element weights equal to 1. On the other hand, it can also highlight the weight of important elements through the inherent mechanism of softmax; (3) the calculation result of the second stage is the corresponding weight coefficient, and then the weighted sum can obtain the attention value.

[0086] Figure 1 A schematic diagram of a system used by an image semantic segmentation method provided by an example embodiment of the present application is shown, as shown in Figure 1 The system includes a server 110 and a terminal 120.

[0087] The server 110 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platform.

[0088] The terminal 120 can be a terminal device with network connection function and data processing function. Further, the terminal can be a terminal with image semantic segmentation function. For example, the terminal 120 can be a smart phone, a tablet computer, an e-book reader, smart glasses, a smart watch, a smart television, an MP3 player, an MP4 player, a laptop computer, a desktop computer, and the like.

[0089] Optionally, the system includes one or more servers 110 and a plurality of terminals 120. The number of servers 110 and terminals 120 is not limited in the embodiments of the present application.

[0090] The terminal and the server are connected through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0091] Optionally, the wireless or wired networks described above use standard communications technologies and / or protocols. The networks typically carry Internet traffic, but can also include, without limitation, any combination of local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), proprietary networks, propriety peer-to-peer communications networks, or virtual networks, including without limitation any combination of fiber optic- based, copper-based, wireless, or other communications media. In some embodiments, data exchanged over the one or more networks is represented using technologies and / or formats including, but not limited to, Hypertext Markup Language (HTML), Extensible Markup Language (XML), and others. In addition, all or some links can be encrypted using conventional encryption technologies, such as, but not limited to, Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), Internet Protocol Security (IPsec), and others. In some embodiments, the data is encrypted using custom and / or proprietary data encryption technologies not described herein. This application is not limited in this regard.

[0092] Figure 2 A flowchart of an image semantic segmentation method is shown, which is shown by an example embodiment of the application, and the method is executed by a computer device, which can be implemented as a terminal or a server, and the terminal or the server can be Figure 1 The terminal or the server is shown as Figure 2 The image semantic segmentation method includes the following steps:

[0093] Step 210, obtaining a target image.

[0094] The target image is an image to be subjected to image semantic segmentation.

[0095] Image semantic segmentation is an important step in image semantic segmentation and computer vision about image understanding, which refers to identifying an image at the pixel level and predicting the class label of each pixel in the image, i.e., labeling the object class to which each pixel in the image belongs.

[0096] Step 220, obtaining a first vector matrix corresponding to the target image in the query dimension, a second vector matrix in the key dimension, and a third vector matrix in the value dimension.

[0097] Wherein, for the self-attention mechanism, the first vector matrix Q (Query), the second vector matrix K (Key) and the third vector matrix V (Value) are all from the same input feature.

[0098] In step 230, the values in the target vector matrix are exchanged n times to obtain an extended vector matrix corresponding to each position exchange; the target vector matrix is at least one of the first vector matrix and the second vector matrix; n is a positive integer.

[0099] In step 240, based on the first vector matrix, the second vector matrix, and the extended vector matrix corresponding to each position exchange, the self-attention vector matrix of the target image is obtained.

[0100] A target image can be regarded as a whole image composed of a plurality of image blocks, and the self-attention vector matrix is used to indicate the correlation between the image blocks in the target image; the value of the self-attention vector matrix depends on the first vector matrix Q (Query) and the second vector matrix K (Key).

[0101] In related technologies, when calculating the self-attention vector matrix, the dot product between Q and K is calculated, and then divided by a scale Wherein, the purpose of dividing by a scale is to prevent the dot product result between Q and K from being too large, D h The vector dimensions of Query and Key are D, and the result is normalized to a probability distribution by using the softmax operation, that is, the calculation process of the self-attention vector matrix A (Q, K) is represented as:

[0102]

[0103] However, due to the inflexible calculation method of calculating the similarity between Q and K in the related art for calculating the self-attention vector matrix, the spatial pixel mismatch occurs when performing semantic segmentation on the image, resulting in inaccurate segmentation of small target objects in the image.

[0104] In the embodiment of the present application, by improving the flexibility of the calculation method of the self-attention vector matrix in calculating the similarity of Q and K, the matching degree of spatial pixels in image semantic segmentation is improved, thereby solving the problem of inaccurate segmentation of small targets in the image. In the embodiment of the present application, when calculating the self-attention vector matrix based on Q and K, the values in the target vector matrix in the first vector matrix and the second vector matrix are exchanged n times to obtain n extended vector matrices corresponding to the target vector matrix, and the self-attention vector matrix of the target image is obtained based on the n extended vectors and the first vector matrix and the second vector matrix, thereby fully exploiting the spatial information contained in different feature dimensions between Q and K.

[0105] The target vector matrix can be one of the first vector matrix and the target vector matrix, or the target vector matrix can also be the first vector matrix and the second vector matrix.

[0106] Step 250, obtaining an output vector matrix based on the self-attention vector matrix and the third vector matrix.

[0107] Step 260, obtaining a semantic segmentation result corresponding to the target image based on the output vector matrix.

[0108] In summary, the image semantic segmentation method provided by the embodiment of the present application, by obtaining the first vector matrix, the second vector matrix, and the third vector matrix corresponding to the target image in Q, K, and V dimensions respectively, the first vector matrix and the target vector matrix in the second vector matrix are exchanged n times to obtain n extended vector matrices, the self-attention vector matrix of the target image is obtained based on the n extended vector matrices, the first vector matrix and the second vector matrix, and the output vector matrix is obtained based on the self-attention vector matrix and the third vector matrix, and the semantic segmentation result of the target image is obtained. In the process of obtaining the semantic segmentation result of the target image, the extended vector matrix can be used to fully exploit the spatial information contained in different feature dimensions between K and Q, thereby improving the accuracy of image semantic segmentation and improving the performance of image semantic segmentation.

[0109] Illustratively, the application scenarios of the image segmentation method provided by the embodiment of the present application include but are not limited to the following scenarios:

[0110] 1) autonomous driving scenario;

[0111] In an autonomous driving scenario, in order to give the vehicle the necessary perception, obtain the environment in which the vehicle is located, so that the autonomous vehicle can drive safely, it is necessary to obtain the real-time road scene image, and perform image semantic segmentation on the road scene image to obtain the position of the obstacle in the road scene, and control the autonomous driving of the vehicle based on the position of the obstacle; in this process, in order to improve the semantic segmentation effect of the road scene image, the image segmentation method provided by the present application can be used, the road scene image obtained by the image acquisition device installed on the vehicle is subjected to image serialization processing, a one-dimensional sequence corresponding to the road scene image is obtained, a first vector matrix in the query dimension, a second vector matrix in the key dimension and a third vector matrix in the value dimension are obtained, the values in the target vector matrix in the first vector matrix and the second vector matrix are subjected to n times of position exchange, and the extended vector matrix corresponding to n times of position exchange is obtained, and based on the first vector matrix, the second vector matrix and the extended vector matrix corresponding to n times of position exchange, the self-attention vector matrix of the road scene image is obtained, and the semantic segmentation result of the road scene image is obtained through the output vector matrix obtained based on the self-attention vector matrix and the third vector matrix, so as to obtain the position of each obstacle in the road scene image; since in the process of obtaining the semantic segmentation result of the road image, the self-attention vector matrix of the road scene image is obtained by combining other vector matrices after the values in the target vector matrix are subjected to multiple cyclic position exchanges, the spatial information between the features of different dimensions is fully mined, thereby improving the accuracy of image semantic segmentation, and further improving the accuracy of obtaining the position of the obstacle in the road scene and the safety of autonomous driving.

[0112] 2) Medical image assisted diagnosis scenario;

[0113] In the medical field, medical images have become an important auxiliary tool for medical assisted diagnosis. Image semantic segmentation is performed on medical images, and different organs in medical images are labeled based on the results of image semantic segmentation, which can better assist medical personnel in medical diagnosis. For example, image semantic segmentation is performed on a chest X-ray film, and the lung and heart regions in the X-ray film are labeled respectively; in this process, the accuracy of image semantic segmentation affects the accuracy of information labeling in medical images, and further affects the accuracy of medical diagnosis based on information labeling by medical personnel. Therefore, in the medical image assisted diagnosis scenario, the image semantic segmentation method provided by the present application can improve the accuracy of semantic segmentation of medical images, and further improve the accuracy of information standard and the accuracy of medical assisted diagnosis.

[0114] The image semantic segmentation method disclosed in the present application can be implemented based on an image semantic segmentation model. The scheme includes an image semantic segmentation model generation stage and an image semantic segmentation stage. Figure 3 is a framework diagram of image semantic segmentation model generation and image semantic segmentation according to an example embodiment of the present application, as shown in Figure 3 In the image semantic segmentation model generation stage, the image semantic segmentation model generation device 310 obtains an image semantic segmentation model through a pre-set training sample data set (including a sample image and an image semantic segmentation label corresponding to the sample image). Then, the image semantic segmentation model is generated based on the image semantic segmentation model. In the image semantic segmentation stage, the image semantic segmentation device 320 processes the input target image based on the image semantic segmentation model to obtain the image semantic result of the target image, such as determining the area and position of each organ in the target medical image.

[0115] The image semantic segmentation model generation device 310 and the image semantic segmentation device 320 can be a computer device, such as a personal computer, a server, or a mobile computer device such as a tablet computer or an e-book reader.

[0116] Optionally, the image semantic segmentation model generation device 310 and the image semantic segmentation device 320 can be the same device, or the image semantic segmentation model generation device 310 and the image semantic segmentation device 320 can be different devices. Moreover, when the image semantic segmentation model generation device 310 and the image semantic segmentation device 320 are different devices, the image semantic segmentation model generation device 310 and the image semantic segmentation device 320 can be the same type of device, such as the image semantic segmentation model generation device 310 and the image semantic segmentation device 320 can both be servers; or the image semantic segmentation model generation device 310 and the image semantic segmentation device 320 can be different types of devices, such as the image semantic segmentation device 320 can be a personal computer or a terminal, and the image semantic segmentation model generation device 310 can be a server. The specific type of the image semantic segmentation model generation device 310 and the image semantic segmentation device 320 is not limited in the present application.

[0117] Figure 4 is a flowchart of a training method of an image semantic segmentation model according to an example embodiment of the present application. The method can be executed by a computer device, which can be implemented as a terminal or a server, such as the terminal or server shown in Figure 1 , as shown in Figure 4 The training method of the image semantic segmentation model includes the following steps:

[0118] At step 410, a sample image set is obtained, the sample image set including at least two sample images and semantic segmentation labels corresponding to the at least two sample images respectively.

[0119] Optionally, the semantic segmentation labels corresponding to the sample images can be annotated by relevant personnel.

[0120] At step 420, the sample image set is used for training to obtain an image semantic segmentation model.

[0121] The image semantic segmentation model can be applied to a terminal or a server, and the image semantic segmentation model is used for semantic segmentation of an input target image to obtain a semantic segmentation result corresponding to the target image.

[0122] Illustratively, the image semantic segmentation model can include a linear projection network (Linear Project), a cycle transformer (Cycle Transformer), and a decoder (Decoder).

[0123] The linear projection network is used to obtain a one-dimensional sequence corresponding to a sample image.

[0124] The cycle transformer is used to obtain, based on the one-dimensional sequence, a first vector matrix in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension corresponding to the sample image; perform n times of position exchange on values in a target vector matrix to obtain n times of extended vector matrices corresponding to the n times of position exchange respectively; obtain a self-attention vector matrix of the sample image based on the first vector matrix, the second vector matrix, and the n times of extended vector matrices corresponding to the n times of position exchange respectively; and obtain an output vector matrix based on the attention vector matrix and the third vector matrix; the target vector matrix is at least one of the first vector matrix and the second vector matrix.

[0125] The decoder is used to obtain a predicted semantic segmentation result corresponding to the sample image based on the output vector matrix.

[0126] In order to facilitate the linear projection network to obtain the one-dimensional sequence corresponding to the sample image, the sample image needs to be preprocessed, and the preprocessing process includes:

[0127] The sample image is divided into N image blocks, N≥2, and N is a positive integer;

[0128] A vector matrix composed of image block vectors corresponding to the N image blocks is obtained; so as to facilitate the linear projection network to perform linear transformation on the vector matrix composed of the N image block vectors to obtain the one-dimensional sequence corresponding to the sample image.

[0129] Illustratively, the training process of the image semantic segmentation model includes:

[0130] obtaining, by the image semantic segmentation model, a predicted semantic segmentation result corresponding to the sample image;

[0131] updating parameters of the image semantic segmentation model based on the predicted semantic segmentation result and the image semantic segmentation label, so as to train the image semantic segmentation model.

[0132] wherein, based on the predicted semantic segmentation result and the image semantic segmentation label, a loss function value can be calculated;

[0133] updating parameters of the image semantic segmentation model based on the loss function value.

[0134] Since the predicted semantic segmentation result obtained by the image semantic segmentation model based on the sample image needs to be the same or similar to the image semantic segmentation label corresponding to the sample image, in order to ensure the accuracy of the image semantic segmentation result of the target image obtained by the image semantic segmentation model, multiple training needs to be performed in the training process of the image semantic segmentation model, and each parameter in each part (linear projection network, cyclic transformer, and decoder) included in the image semantic segmentation model needs to be updated until the image semantic segmentation model converges.

[0135] In one possible implementation, the cyclic transformer includes a multi-head cyclic self-attention layer; the multi-head cyclic self-attention layer is configured to obtain x vector matrix groups in parallel based on x sets of weight matrices, each vector matrix group including a first vector sub-matrix, a second vector sub-matrix, and a third vector sub-matrix corresponding thereto; the first vector matrix includes the first vector sub-matrix included in each of the x vector matrix groups, the second vector matrix includes the second vector sub-matrix included in each of the x vector matrix groups, and the third vector matrix includes the third vector sub-matrix included in each of the x vector matrix groups; self-attention vector sub-matrices corresponding to the x vector matrix groups are obtained in parallel; and an output vector matrix is obtained based on the self-attention vector sub-matrices corresponding to the x vector matrix groups and the third vector matrix.

[0136] That is, the image semantic segmentation model of the present application is a model constructed based on a multi-head attention mechanism, wherein a plurality of sets of (Q, K, V) matrices are included in the multi-head attention mechanism, and a set of (Q, K, V) matrices represents an operation of a self-attention mechanism once. After the plurality of matrices are spliced and multiplied by a projection matrix, the output of the final multi-head attention layer, i.e., the output vector matrix in the present application, can be obtained.

[0137] Figure 5A schematic diagram of an image semantic segmentation model according to an example embodiment of the present application is shown in FIG. 5, which includes a linear projection network 510, a cyclic Transformer 520, and a decoder 530. Optionally, the cyclic Transformer is composed of a multi-head cycle self-attention layer, a multi-layer perceptron, and a layer normalization layer. Figure 5

[0138] The image semantic segmentation model includes L layers of cyclic Transformers, and the value of L can be determined based on the amount of task data. The amount of task data includes the size of the target image to be processed and the number of target images to be processed. The larger the value of L, the deeper the number of stacked cyclic Transform layers. The smaller the value of L, the shallower the number of stacked cyclic Transform layers. Illustratively, the number of cyclic Transform layers can be 12.

[0139] The image semantic segmentation model provided by the present application is a general framework that can be embedded into any Transformer framework, and has strong versatility.

[0140] The image semantic segmentation model obtains the image semantic segmentation result corresponding to the target image. The process of obtaining the image semantic segmentation result corresponding to the target image based on the image semantic segmentation model can be implemented as follows:

[0141] Obtaining a target image;

[0142] Obtaining a first vector matrix in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension of the target image through a cyclic Transformer in the image semantic segmentation model; performing n times of position exchange on the values in the target vector matrix to obtain n times of extended vector matrices corresponding to the position exchange, respectively; obtaining a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the n times of extended vector matrices corresponding to the position exchange, respectively; and obtaining an output vector matrix based on the attention vector matrix and the third vector matrix. The target vector matrix is at least one of the first vector matrix and the second vector matrix.

[0143] Obtaining a semantic segmentation result corresponding to the target image based on the output vector matrix through a decoder in the image semantic segmentation model.

[0144] ​The image semantic segmentation method and the training method of the image semantic segmentation model provided in the application can be used in combination with the blockchain technology. Illustratively, the target image to be subjected to image semantic segmentation or the sample image used to train the image semantic segmentation model can be stored in the blockchain system, and the relevant image can be obtained from the blockchain system when the image semantic segmentation or the training of the image semantic segmentation model is performed. Meanwhile, the semantic segmentation result of the image can also be stored in the blockchain system to ensure the security of the data.

[0145] To sum up, the training method of the image semantic segmentation model provided in the application trains the image semantic segmentation model including the linear projection network, the cyclic Transformer, and the decoder based on the sample image set, so that when the image semantic segmentation result of the target image is extracted, the first vector matrix, the second vector matrix, and the third vector matrix corresponding to the target image in the Q, K, and V dimensions are obtained, the target vector matrix in the first vector matrix and the second vector matrix is subjected to n times of position exchange to obtain n extended vector matrices, the self-attention vector matrix of the target image is obtained based on the n extended vector matrices, the first vector matrix, and the second vector matrix, and the output vector matrix is obtained based on the self-attention vector matrix and the third vector matrix, and the semantic segmentation result of the target image is obtained. In the process of obtaining the semantic segmentation result of the target image, the extended vector matrix can be used to fully mine the spatial information contained in the different feature dimensions between K and Q, thereby improving the accuracy of the image semantic segmentation and improving the performance of the image semantic segmentation.

[0146] The image semantic segmentation method provided in the embodiment of the application is implemented based on the image semantic segmentation model, and the image semantic segmentation model is a model built based on the cyclic Transformer, Figure 6 A flowchart of the image semantic segmentation method provided in an example embodiment of the application is shown. The method can be executed by a computer device, which can be implemented as a terminal or a server. The terminal or the server can be a mobile phone, a tablet computer, a personal computer, a server, or the like. Figure 1 As shown in the terminal or the server, Figure 6 The image semantic segmentation method includes the following steps:

[0147] Step 610: Obtain a target image.

[0148] Step 620: Perform serialization processing on the target image to obtain a one-dimensional sequence corresponding to the target image.

[0149] Optionally, the linear projection network in the image semantic segmentation model is used to perform serialization processing on the target image to obtain the one-dimensional sequence corresponding to the target image.

[0150] The process of obtaining the one-dimensional sequence corresponding to the target image can be implemented as follows:

[0151] The target image is divided into N image blocks, N≥2, and N is a positive integer;

[0152] A vector matrix composed of image block vectors corresponding to the N image blocks is obtained;

[0153] The vector matrix composed of the N image block vectors is linearly transformed to obtain the one-dimensional sequence corresponding to the target image.

[0154] In a possible implementation, the linear projection module can also obtain the one-dimensional sequence corresponding to the target image through a 1*1 convolution operation.

[0155] Since the input of the recurrent Transformer in the image semantic segmentation model is a one-dimensional sequence, before inputting the target image into the recurrent Transformer, the target image (H*W*C) needs to be converted into a one-dimensional sequence, where H (height) represents the height of the target image, W (width) represents the width of the target image, and C (channel) represents the number of channels of the target image; taking the number of channels of the target image as 3 as an example, for a target image x∈R H×W×3 , first, the target image is divided into p*q image blocks, where p and q represent the height and width of the image block respectively, and the values of p and q of the same image block can be the same or different, and the p and q corresponding to different image blocks can correspond to the same or different. Assuming that the p and q corresponding to different image blocks are the same, the number of image blocks N can be denoted as:

[0156]

[0157] Secondly, the image block vector corresponding to each image block is obtained, and then N image block vectors with a dimension of 3*p*q can be obtained, which are used to indicate the pixel features of each image block; the vectors are stacked together to obtain a vector matrix X∈R 3×p×q×N composed of the N image block vectors. The vector matrix composed of the N image block vectors is linearly transformed to obtain the one-dimensional sequence corresponding to the target image; wherein the linear transformation is a feature space conversion of the N image block vectors, and the conversion matrix is The linear transformation process is represented as:

[0158]

[0159] Where Y represents the one-dimensional sequence corresponding to the target image; the above process realizes the conversion of the target image from two-dimensional to one-dimensional sequence signal, which facilitates the subsequent recurrent Transformer to learn features and integrate information.

[0160] At step 630, based on the one-dimensional sequence, a first vector matrix corresponding to the target image in the query dimension, a second vector matrix in the key dimension, and a third vector matrix in the value dimension are obtained.

[0161] The process can be represented as:

[0162]

[0163] wherein D represents the dimension of the output feature, D h represents the dimension of the input feature, U QKV is a learnable parameter matrix in the recurrent Transformer, which can be updated in model training.

[0164] The first vector matrix, the second vector matrix, and the third vector matrix respectively contain feature vectors corresponding to each image block in the target image.

[0165] At step 640, the values in the target vector matrix are exchanged n times in units of feature vectors to obtain extended vector matrices corresponding to the n times of position exchange respectively.

[0166] In one possible implementation, the n times of position exchange of the values in the target vector matrix can be implemented as:

[0167] The n times of position exchange of the values in the target vector matrix are randomly exchanged in units of feature vectors to obtain extended vector matrices corresponding to the n times of position exchange respectively.

[0168] That is, when the values in the target vector matrix are exchanged, the feature vectors for each position exchange are not limited, and the positions of the feature vectors in the feature vector queue after position exchange are also not limited. On the premise that the feature vector queue after position exchange is different from that before position exchange, the values in the target vector matrix are randomly exchanged n times.

[0169] Alternatively, in another possible implementation, the n times of cyclic position exchange of the values in the target vector matrix are exchanged in units of feature vectors according to a specified step to obtain extended vector matrices corresponding to the n times of cyclic position exchange respectively.

[0170] The value of the specified step can be set by relevant personnel based on actual needs. The cyclic position exchange refers to adjusting the positions of the feature vectors in the feature vector queue corresponding to the target vector matrix according to a certain position exchange order. Illustratively, the positions of the feature vectors can be adjusted in the order from the head to the tail, for example, in one position exchange, the first specified number of feature vectors at the head are moved to the tail. Alternatively, the positions of the feature vectors can be adjusted in the order from the tail to the head. The process can be implemented as:

[0171] Obtain the queue of eigenvectors that make up the target vector matrix;

[0172] In response to a cyclic position swap, the feature vector at the end of the feature vector queue with a specified step size is moved to the beginning of the feature vector queue, and other feature vectors in the feature vector queue are moved sequentially.

[0173] Specifically, when moving feature vectors with a specified step size, the current arrangement of the feature vectors with the specified step size can be maintained, and the feature vectors with the specified step size can be moved as a whole to the beginning of the feature vector queue. For example, if the specified step size is 2, the vector matrix corresponding to the feature vector queue is [1, 2, 3, 4]. After one cycle of position swapping, the vector matrix corresponding to the feature vector queue is [3, 4, 1, 2]. Alternatively, feature vectors with the specified step size can be moved sequentially to the beginning of the feature vector queue according to their current arrangement from front to back. For example, if the specified step size is 2, the vector matrix corresponding to the feature vectors is [1, 2, 3, 4]. After one cycle of position swapping, the vector matrix corresponding to the feature vector queue is [4, 3, 1, 2].

[0174] Indicatively, when the specified step size is 1, during a single loop position swap, the value at the last position in the feature vector queue is moved to the first position in the feature vector queue, and the other values ​​in the feature vector queue are moved sequentially.

[0175] The vector matrix obtained after one loop of position operations is converted into an extended vector matrix.

[0176] Figure 7 This is a schematic diagram illustrating a cyclic position exchange in an exemplary embodiment of this application, as shown below. Figure 7 As shown, taking a specified step size of 1 and performing cyclic swaps in order from the tail to the head as an example, the feature vector queue of the target vector matrix contains 4 feature vectors. Before the cyclic swaps, the vector matrix corresponding to this feature vector queue is [1, 2, 3, 4]; after one cyclic swap, the vector matrix corresponding to this feature vector queue is [4, 1, 2, 3]; after two cyclic swaps, the vector matrix corresponding to this feature vector queue is [3, 4, 1, 2]; after three cyclic swaps, the vector matrix corresponding to this feature vector queue is [2, 3, 4, 1].

[0177] Since the number of eigenvectors in the eigenvector queue corresponding to the target vector matrix is ​​finite, as the cyclic position swaps proceed, the resulting eigenvector queue will revert to its state before any cyclic position swaps were performed after a certain cyclic position swap. For example, ... Figure 7The target vector matrix shown in the figure, after four times of cyclic position exchange, the vector matrix corresponding to the eigenvector queue is [1, 2, 3, 4], which is the same as the state before any cyclic position operation is performed on the target vector matrix, and the vector matrix obtained by subsequent cyclic position operations will be repeated, thereby repeated calculation occurs, causing redundancy. Therefore, in a possible implementation manner, in response to the number of cyclic position exchanges reaching a number threshold, the cyclic position exchange is stopped; the number threshold is determined based on the number of eigenvectors in the target vector matrix and a specified step length; illustratively, the number threshold = the number of eigenvectors in the target vector matrix / the specified step length - 1. For example, when the specified step length is 1 and the number of eigenvectors in the target vector matrix is 4, the number threshold = 3.

[0178] Alternatively, in another possible implementation manner, in response to the arrangement of the eigenvectors in the eigenvector queue being the same as the original numerical arrangement after one cyclic position exchange, the cyclic position exchange is stopped, and the original numerical arrangement refers to the arrangement of the eigenvectors in the eigenvector queue before the cyclic position exchange is performed.

[0179] In step 650, based on the first vector matrix, the second vector matrix, and the extended vector matrix corresponding to n times of position exchange respectively, the self-attention vector matrix of the target image is obtained.

[0180] In the embodiments of the present application, the target vector matrix can include the first vector matrix and the second vector matrix, that is, the position exchange operation can be performed on the first vector matrix and the second vector matrix at the same time, and correspondingly, the number of position exchanges n is the sum of the number of position exchanges of the first vector matrix and the number of position exchanges of the second vector matrix, and the extended vector matrix corresponding to n times of position exchange respectively includes the extended vector matrix after the position exchange of the first vector matrix and the extended vector matrix after the position exchange of the second vector matrix. The self-attention vector matrix of the target image is obtained based on the extended vector matrix corresponding to the first vector matrix and the second vector matrix together and the first vector matrix and the second vector matrix.

[0181] Alternatively, the target vector matrix can be implemented as the first vector matrix, or the target vector matrix is implemented as the second vector matrix; when the target vector matrix is the second vector matrix, the process of calculating the attention matrix of the target image is as follows:

[0182] The n point multiplication results of the first vector matrix and the extended vector matrix corresponding to n times of position exchange are obtained;

[0183] The average value of the n point multiplication results is obtained;

[0184] Based on the average value, the self-attention vector matrix of the target image is obtained.

[0185] The process can be represented as:

[0186]

[0187] wherein A represents a self-attention vector matrix, or an attention map, Cycle represents a cycle operation, and C=n represents the number of cycles.

[0188] When the target vector matrix is the first vector matrix, the process of calculating the attention matrix of the target image can be represented as:

[0189]

[0190] At step 660, an output vector matrix is obtained based on the self-attention vector matrix and the third vector matrix.

[0191] When the cycle Transformer includes a cycle attention layer, the process of calculating the output vector matrix SA(z) is represented as:

[0192] SA(z) = A cycle V

[0193] When the cycle Transformer includes a multi-head cycle self-attention layer, each cycle self-attention layer corresponds to obtain a self-attention vector sub-matrix and a plurality of product results of the self-attention vector sub-matrix and the third vector matrix. Therefore, the plurality of product results need to be further processed to obtain the output vector matrix. The process of obtaining the output vector matrix of the multi-head cycle self-attention layer is implemented as:

[0194] Based on the x sets of weight matrices, x vector matrix groups are obtained, each of which includes a first vector sub-matrix, a second vector sub-matrix, and a third vector sub-matrix corresponding thereto. The first vector sub-matrix in the first vector matrix includes the first vector sub-matrix included in each of the x vector matrix groups, the second vector sub-matrix in the second vector matrix includes the second vector sub-matrix included in each of the x vector matrix groups, and the third vector sub-matrix in the third vector matrix includes the third vector sub-matrix included in each of the x vector matrix groups. x is a positive integer;

[0195] The values in the target vector sub-matrix corresponding to each of the x vector matrix groups are exchanged n times to obtain an extended vector sub-matrix corresponding to each of the n times of position exchange. The target vector sub-matrix is at least one of the first vector sub-matrix and the second vector sub-matrix. The extended vector matrix includes the extended vector sub-matrix corresponding to each of the x vector matrix groups.

[0196] The self-attention vector sub-matrix corresponding to each of the x vector matrix groups is obtained based on the first vector sub-matrix, the second vector sub-matrix corresponding to each of the x vector matrix groups, and the extended vector sub-matrix corresponding to the n-time position exchange.

[0197] The output vector matrix is obtained based on the x self-attention vector sub-matrices and the third vector matrix.

[0198] The x sets of weight matrices are obtained after the x cyclic self-attention layers are updated based on x different random matrices during the model training process, and the x cyclic self-attention layers are processed in parallel during the obtaining of the self-attention vector sub-matrix. The process of obtaining the self-attention vector sub-matrix by each cyclic self-attention layer can refer to the related content of steps 630 to 650, which will not be described here.

[0199] The output vector matrix is obtained based on the x self-attention vector sub-matrices and the third vector matrix.

[0200] The product result obtained by multiplying the self-attention vector sub-matrix of the target image corresponding to each vector matrix group and the third vector matrix is obtained.

[0201] The x product results are concatenated to obtain a concatenated result.

[0202] The output vector matrix is obtained by projecting the concatenated result.

[0203] The calculation process of the output vector matrix can be represented as:

[0204]

[0205] Wherein, U msa The parameter of the projection matrix can be updated during the model training process.

[0206] The steps 630 to 660 described above can be executed by the cyclic transformer in the image semantic model.

[0207] Step 670, based on the output vector matrix, the semantic segmentation result corresponding to the target image is obtained.

[0208] The process can be executed by the decoder in the image semantic segmentation model. The decoding of the output vector matrix is to decode the features converged by the cyclic transformer in the control and channel dimensions to obtain the final semantic segmentation result.

[0209] Among them, the commonly used decoding methods are direct upsampling and 1x1 convolution operation, multi-scale fusion upsampling and progressive upsampling. Taking the progressive upsampling method as an example, Figure 8 A schematic diagram of a decoder is shown according to an example embodiment of the present application, as shown in Figure 8 As shown, after the last loop of the Transformer feature, the feature is re-exhibited as a Tensor, and then gradually recovered to the spatial size of the input image through 2x upsampling and convolution combination, and the channel number is the class number of the entire data set, and finally the semantic segmentation result of the target image is output.

[0210] In summary, the image semantic segmentation method provided by the embodiments of the present application obtains the first vector matrix, the second vector matrix and the third vector matrix corresponding to the target image in the Q, K and V dimensions, respectively, and then performs n times of position exchange on the first vector matrix and the target vector matrix based on the second vector matrix to obtain n extended vector matrices, obtains the self-attention vector matrix of the target image based on the n extended vector matrices, the first vector matrix and the second vector matrix, and obtains the semantic segmentation result of the target image based on the output vector matrix obtained based on the self-attention vector matrix and the third vector matrix. In the process of obtaining the semantic segmentation result of the target image, the extended vector matrix can be used to fully exploit the spatial information contained in different feature dimensions between K and Q, thereby improving the accuracy of image semantic segmentation and improving the performance of image semantic segmentation.

[0211] Figure 9 A block diagram of an image semantic segmentation device is shown according to an example embodiment of the present application, as shown in Figure 9 As shown, the device includes:

[0212] The target image acquisition module 910 is configured to acquire a target image.

[0213] The first acquisition module 920 is configured to acquire a first vector matrix corresponding to the target image in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension.

[0214] The position exchange module 930 is configured to perform n times of position exchange on the values in the target vector matrix to obtain n extended vector matrices corresponding to the n times of position exchange, respectively; the target vector matrix is at least one of the first vector matrix and the second vector matrix.

[0215] The second acquisition module 940 is configured to acquire a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the n extended vector matrices corresponding to the n times of position exchange, respectively.

[0216] The third obtaining module 950 is configured to obtain an output vector matrix based on the attention vector matrix and the third vector matrix.

[0217] The semantic segmentation result obtaining module 960 is configured to obtain a semantic segmentation result corresponding to the target image based on the output vector matrix.

[0218] In a possible implementation, the first vector matrix, the second vector matrix, and the third vector matrix respectively include feature vectors of respective image blocks in the target image.

[0219] The position exchange module 930 is configured to perform n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices respectively corresponding to the n times of position exchange.

[0220] In a possible implementation, the position exchange module 930 is configured to perform n times of cyclic position exchange on values in the target vector matrix in units of the feature vectors according to a specified step length to obtain the extended vector matrices respectively corresponding to the n times of cyclic position exchange.

[0221] In a possible implementation, the position exchange module 930 includes:

[0222] The queue obtaining sub-module is configured to obtain a feature vector queue composed of the feature vectors of the target vector matrix.

[0223] The position exchange sub-module is configured to, in response to one time of cyclic position exchange, move the feature vectors with the specified step length at the end of the feature vector queue to the beginning of the feature vector queue, and sequentially move other feature vectors in the feature vector queue.

[0224] In a possible implementation, the position exchange sub-module is further configured to, in response to a number of times of cyclic position exchange reaching a threshold number of times, stop the cyclic position exchange, where the threshold number of times is determined based on a number of the feature vectors in the target vector matrix and the specified step length.

[0225] In a possible implementation, the position exchange module 930 is configured to randomly perform n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices respectively corresponding to the n times of position exchange.

[0226] In a possible implementation, the second obtaining module 940 includes:

[0227] The first obtaining sub-module is configured to obtain n point multiplication results of the first vector matrix and the extended vector matrices respectively corresponding to the n times of position exchange.

[0228] a second obtaining sub-module, configured to obtain an average value of the n point multiplication results;

[0229] a third obtaining sub-module, configured to obtain a self-attention vector matrix of the target image based on the average value.

[0230] In a possible implementation, the apparatus further includes:

[0231] a serialization processing module, configured to perform serialization processing on the target image to obtain a one-dimensional sequence corresponding to the target image, before obtaining a first vector matrix corresponding to the target image in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension.

[0232] The first obtaining module 920 is configured to obtain the first vector matrix, the second vector matrix, and the third vector matrix of the target image based on the one-dimensional sequence.

[0233] In a possible implementation, the serialization processing module includes:

[0234] an image division sub-module, configured to divide the target image into N image blocks, where N is greater than or equal to 2 and is a positive integer.

[0235] a fourth obtaining sub-module, configured to obtain a vector matrix composed of image block vectors corresponding to the N image blocks.

[0236] a linear transformation sub-module, configured to perform linear transformation on the vector matrix composed of the N image block vectors to obtain the one-dimensional sequence corresponding to the target image.

[0237] In a possible implementation, the first obtaining module 920 is configured to obtain x different vector matrix groups based on x groups of weight matrices, where each of the vector matrix groups includes a respective first vector sub-matrix, a respective second vector sub-matrix, and a respective third vector sub-matrix; the first vector matrix includes first vector sub-matrices included in the x vector matrix groups; the second vector matrix includes second vector sub-matrices included in the x vector matrix groups; the third vector matrix includes third vector sub-matrices included in the x vector matrix groups; x is a positive integer; and x is a positive integer.

[0238] The position exchange module 930 is configured to perform n times of position exchange on values in target vector submatrices corresponding to x vector matrix groups respectively, to obtain n times of extended vector submatrices corresponding to the n times of position exchange respectively; the target vector submatrices are at least one of the first vector submatrix and the second vector submatrix; the extended vector matrix includes x extended vector submatrices corresponding to the x vector matrix groups respectively;

[0239] The second acquisition module 940 is configured to acquire self-attention vector submatrices of the target images corresponding to the x vector matrix groups respectively based on the first vector submatrices, the second vector submatrices, and the extended vector submatrices corresponding to the n times of position exchange respectively corresponding to the x vector matrix groups respectively; the self-attention vector matrix includes self-attention vector submatrices corresponding to the x vector matrix groups respectively;

[0240] The third acquisition module 950 is configured to acquire the output vector matrix based on the self-attention vector submatrices of the target images corresponding to the x vector matrix groups respectively and the third vector matrix.

[0241] In a possible implementation, the third acquisition module 950 includes:

[0242] A product result acquisition submodule is configured to acquire x product results obtained by multiplying the self-attention vector submatrices of the target images corresponding to the x vector matrix groups respectively and the third vector matrix;

[0243] A concatenation submodule is configured to concatenate the x product results to obtain a concatenation result.

[0244] A projection submodule is configured to project the concatenation result to obtain the output vector matrix.

[0245] In summary, the image semantic segmentation device provided by the embodiment of the present application obtains the first vector matrix, the second vector matrix, and the third vector matrix corresponding to the target image in the Q, K, and V dimensions respectively, then performs n times of position exchange on the target vector submatrices in the first vector submatrix and the second vector submatrix to obtain n extended vector submatrices, acquires the self-attention vector submatrices of the target images based on the n extended vector submatrices, the first vector submatrix, and the second vector submatrix, and acquires the output vector matrix based on the self-attention vector submatrices and the third vector matrix, to obtain the semantic segmentation result of the target image. In the process of acquiring the semantic segmentation result of the target image, the extended vector submatrices can be used to fully mine the spatial information contained in different feature dimensions between K and Q, thereby improving the accuracy of image semantic segmentation and improving the performance of image semantic segmentation.

[0246] Figure 10 A block diagram of an image semantic segmentation device according to an example embodiment of the present application is shown in FIG. 1. Figure 10 As shown, the device includes:

[0247] A target image acquisition module 1010 is configured to acquire a target image.

[0248] An output vector matrix acquisition module 1020 is configured to acquire, by a cyclic transformer in an image semantic segmentation model, a first vector matrix corresponding to the target image in a query dimension, a second vector matrix corresponding to the target image in a key dimension, and a third vector matrix corresponding to the target image in a value dimension; perform n times of position exchange on values in the target vector matrix to obtain n extended vector matrices corresponding to the n times of position exchange, respectively; acquire a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the extended vector matrices corresponding to the n times of position exchange, respectively; and acquire an output vector matrix based on the attention vector matrix and the third vector matrix; the target vector matrix is at least one of the first vector matrix and the second vector matrix.

[0249] A semantic segmentation result acquisition module 1030 is configured to acquire, by a decoder in the image semantic segmentation model, a semantic segmentation result corresponding to the target image based on the output vector matrix.

[0250] In summary, the image semantic segmentation device provided by the example embodiments of the present application can obtain a first vector matrix, a second vector matrix, and a third vector matrix corresponding to a target image in Q, K, and V dimensions, respectively, perform n times of position exchange on a first vector matrix and a target vector matrix based on a second vector matrix to obtain n extended vector matrices, acquire a self-attention vector matrix of the target image based on the n extended vector matrices, the first vector matrix, and the second vector matrix, and acquire an output vector matrix based on the self-attention vector matrix and the third vector matrix to obtain a semantic segmentation result of the target image. In the process of acquiring the semantic segmentation result of the target image, the extended vector matrices can be used to fully exploit spatial information contained in different feature dimensions between K and Q, thereby improving the accuracy and performance of image semantic segmentation.

[0251] Figure 11A structural block diagram of a computer device 1100 is shown according to an example embodiment of the present application. The computer device can be implemented as a server in the above-mentioned solutions of the present application. The computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 also includes a mass storage device 1106 for storing an operating system 1109, application programs 1110, and other program modules 1111.

[0252] The mass storage device 1106 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1106 and its associated computer readable media provide nonvolatile storage for the computer device 1100. That is, the mass storage device 1106 can include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0253] Without loss of generality, the computer readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read only memory (EPROM), electrically-erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, digital versatile discs (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. Of course, the computer storage media is not limited to the above-mentioned several kinds. The system memory 1104 and the mass storage device 1106 mentioned above can be collectively referred to as memory.

[0254] According to various embodiments of the present disclosure, the computer device 1100 can also operate connected to a network, such as the Internet, by a network connection. That is, the computer device 1100 can be connected to a network 1108 through a network interface unit 1107 connected to the system bus 1105, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 1107.

[0255] The memory further includes at least one instruction, at least one program, a code set or an instruction set stored in the memory, and the central processing unit 1101 implements all or part of the steps of the image semantic segmentation method shown in each of the embodiments by executing the at least one instruction, at least one program, code set or instruction set.

[0256] Figure 12 A structural block diagram of a computer device 1200 provided by an example embodiment of the present application is shown. The computer device 1200 can be implemented as the face quality evaluation device and / or the quality evaluation model training device described above, such as a smart phone, a tablet computer, a notebook computer or a desktop computer. The computer device 1200 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, and other names.

[0257] Generally, the computer device 1200 includes a processor 1201 and a memory 1202.

[0258] The processor 1201 can include one or more processing cores, such as a 4-core processor, a 12-core processor, and the like. The processor 1201 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1201 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 1201 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0259] The memory 1202 can include one or more computer-readable storage media that can be non-transitory. The memory 1202 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one instruction for being executed by the processor 1201 to implement the image semantic segmentation method provided by the method embodiment of the present application.

[0260] In some embodiments, the computer device 1200 can also optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202, and the peripheral device interface 1203 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1203 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, a positioning assembly 1208, and a power supply 1209.

[0261] The peripheral interface 1203 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202 and the peripheral interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202 and the peripheral interface 1203 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0262] In some embodiments, the computer device 1200 further comprises one or more sensors 1210. The one or more sensors 1210 include, but are not limited to, an acceleration sensor 1211, a gyroscope sensor 1212, a pressure sensor 1213, a fingerprint sensor 1214, an optical sensor 1215, and a proximity sensor 1216.

[0263] Those skilled in the art can understand that the structure shown in the above Figure 12 The structure shown in the above

[0264] In an exemplary embodiment, a computer readable storage medium is also provided for storing at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement all or part of the steps of the above image semantic segmentation method. For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0265] In an exemplary embodiment, a computer program product or computer program is also provided, which comprises computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes all or part of the steps of the above Figure 2 、 Figure 4 or Figure 6 any embodiment.

[0266] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0267] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the drawings, and that various changes in shape, size and arrangements of parts can be made without departing from the scope of the application. The scope of the application is only limited by the appended claims.

Claims

1. An image semantic segmentation method, characterized in that, The method comprises: acquiring a target image; acquiring a first vector matrix corresponding to the target image on a query dimension, a second vector matrix on a key dimension, and a third vector matrix on a value dimension; performing n times of position exchange on values in a target vector matrix to obtain extended vector matrices corresponding to the n times of position exchange respectively; the target vector matrix is at least one of the first vector matrix and the second vector matrix; n is a positive integer; acquiring a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the extended vector matrices corresponding to the n times of position exchange respectively; acquiring an output vector matrix based on the self-attention vector matrix and the third vector matrix; acquiring a semantic segmentation result corresponding to the target image based on the output vector matrix.

2. The method of claim 1, wherein, The first vector matrix, the second vector matrix, and the third vector matrix respectively comprise feature vectors of each image block in the target image; The performing n times of position exchange on values in a target vector matrix to obtain extended vector matrices corresponding to the n times of position exchange respectively comprises: performing n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices corresponding to the n times of position exchange respectively.

3. The method of claim 2, wherein, The performing n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices corresponding to the n times of position exchange respectively comprises: performing n times of cyclic position exchange on values in the target vector matrix in units of the feature vectors according to a specified step length to obtain the extended vector matrices corresponding to the n times of cyclic position exchange respectively.

4. The method of claim 3, wherein, The performing n times of cyclic position exchange on values in the target vector matrix in units of the feature vectors according to a specified step length to obtain the extended vector matrices corresponding to the n times of cyclic position exchange respectively comprises: acquiring a feature vector queue constituting the target vector matrix; in response to one time of cyclic position exchange, moving the feature vectors of the specified step length at the end of the feature vector queue to the beginning of the feature vector queue, and sequentially moving other feature vectors in the feature vector queue.

5. The method of claim 4, wherein, The method further comprises: in response to the number of times of cyclic position exchange reaching a threshold number of times, stopping the cyclic position exchange; the threshold number of times is determined based on the number of the feature vectors in the target vector matrix and the specified step length.

6. The method of claim 2, wherein, The performing n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices corresponding to the n times of position exchange respectively comprises: randomly performing n times of position exchange on values in the target vector matrix in units of the feature vectors to obtain the extended vector matrices corresponding to the n times of position exchange respectively.

7. The method of claim 1, wherein, In response to the target vector matrix being the second vector matrix, the acquiring a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the extended vector matrices corresponding to the n times of position exchange respectively comprises: obtaining n point multiplication results of the first vector matrix and the extended vector matrix corresponding to n times of position exchange respectively; obtaining an average value of the n point multiplication results; obtaining a self-attention vector matrix of the target image based on the average value.

8. The method of claim 1, wherein, Before obtaining the first vector matrix corresponding to the target image in the query dimension, the second vector matrix corresponding to the target image in the key dimension, and the third vector matrix corresponding to the target image in the value dimension, the method further comprises: serializing the target image to obtain a one-dimensional sequence corresponding to the target image; the obtaining the first vector matrix corresponding to the target image in the query dimension, the second vector matrix corresponding to the target image in the key dimension, and the third vector matrix corresponding to the target image in the value dimension comprises: obtaining the first vector matrix, the second vector matrix, and the third vector matrix of the target image based on the one-dimensional sequence.

9. The method of claim 8, wherein, the serializing the target image to obtain a one-dimensional sequence corresponding to the target image comprises: dividing the target image into N image blocks, N≥2, and N is a positive integer; obtaining a vector matrix composed of image block vectors corresponding to the N image blocks; performing linear transformation on the vector matrix composed of the N image block vectors to obtain the one-dimensional sequence corresponding to the target image.

10. The method of claim 1, wherein, the obtaining the first vector matrix corresponding to the target image in the query dimension, the second vector matrix corresponding to the target image in the key dimension, and the third vector matrix corresponding to the target image in the value dimension comprises: based on x sets of weight matrices, obtaining x vector matrix groups, each of the vector matrix groups containing a corresponding first vector sub-matrix, a second vector sub-matrix, and a third vector sub-matrix; the first vector matrix contains the first vector sub-matrix contained in each of the x vector matrix groups, the second vector matrix contains the second vector sub-matrix contained in each of the x vector matrix groups, and the third vector matrix contains the third vector sub-matrix contained in each of the x vector matrix groups; x is a positive integer; the performing n times of position exchange on the values in the target vector matrix to obtain an extended vector matrix corresponding to each of the n times of position exchange comprises: performing n times of position exchange on the values in the target vector sub-matrix corresponding to each of the x vector matrix groups to obtain an extended vector sub-matrix corresponding to each of the n times of position exchange; the target vector sub-matrix is at least one of the first vector sub-matrix and the second vector sub-matrix; the extended vector matrix contains the extended vector sub-matrix corresponding to each of the x vector matrix groups; the obtaining the self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the extended vector matrix corresponding to each of the n times of position exchange comprises: obtaining a self-attention vector sub-matrix of the target image corresponding to each of the x vector matrix groups based on the first vector sub-matrix, the second vector sub-matrix, and the extended vector sub-matrix corresponding to each of the n times of position exchange corresponding to each of the x vector matrix groups; the self-attention vector matrix contains the self-attention vector sub-matrix corresponding to each of the x vector matrix groups. ​ The output vector matrix is obtained based on the attention vector matrix and the third vector matrix. The output vector matrix is obtained based on the x self-attention vector sub-matrices and the third vector matrix.

11. The method of claim 10, wherein, The output vector matrix is obtained based on the x self-attention vector sub-matrices and the third vector matrix. The product result of the multiplication of the self-attention vector sub-matrix of the target image corresponding to each vector matrix group and the third vector matrix is obtained. The x product results are concatenated to obtain a concatenated result. The concatenated result is projected to obtain the output vector matrix.

12. An image semantic segmentation method, characterized in that, The method comprises: obtaining a target image; obtaining, by a recurrent Transformer in an image semantic segmentation model, a first vector matrix corresponding to the target image in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension; performing n times of position exchange on values in a target vector matrix to obtain extended vector matrices corresponding to the n times of position exchange respectively; obtaining a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the extended vector matrices corresponding to the n times of position exchange respectively; obtaining an output vector matrix based on the attention vector matrix and the third vector matrix; the target vector matrix is at least one of the first vector matrix and the second vector matrix; obtaining, by a decoder in the image semantic segmentation model, a semantic segmentation result corresponding to the target image based on the output vector matrix.

13. An image semantic segmentation apparatus, characterized by comprising: The device comprises: a target image obtaining module configured to obtain a target image; a first obtaining module configured to obtain a first vector matrix corresponding to the target image in a query dimension, a second vector matrix in a key dimension, and a third vector matrix in a value dimension; a position exchange module configured to perform n times of position exchange on values in a target vector matrix to obtain extended vector matrices corresponding to the n times of position exchange respectively; the target vector matrix is at least one of the first vector matrix and the second vector matrix; a second obtaining module configured to obtain a self-attention vector matrix of the target image based on the first vector matrix, the second vector matrix, and the extended vector matrices corresponding to the n times of position exchange respectively; a third obtaining module configured to obtain an output vector matrix based on the attention vector matrix and the third vector matrix; a semantic segmentation result obtaining module configured to obtain a semantic segmentation result corresponding to the target image based on the output vector matrix.

14. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set, or an instruction set, which are loaded and executed by the processor to implement the image semantic segmentation method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, and the computer program is loaded and executed by the processor to implement the image semantic segmentation method according to any one of claims 1 to 12.

Images