Image processing method and device, equipment and medium

Abstract

The embodiment of the invention discloses an image processing method and device, equipment and a medium, and is applied to the technical field of image processing. The method comprises the following steps: acquiring a target image containing a target object, an image feature vector and an object query vector, generating an object feature vector based on the object query vector and the image feature vector, and acquiring a category coding vector corresponding to each object category in an object category set, and determining the similarity between the object feature vector and the category coding vector corresponding to each object category so as to determine the object category to which the target object belongs from the object category set. According to the embodiment of the invention, the accuracy of object segmentation in the image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image processing method and device, equipment and a medium. BACKGROUND

[0002] At present, object segmentation (also known as object division) of an image is an important task in the field of image processing. The "segmentation" refers to identifying a target object (such as a vehicle, an airplane, etc.) from an image and determining the position of the target object in the image. The existing object segmentation method is usually to input an image into a pre-trained segmentation model, and the segmentation model detects and identifies a corresponding target object in the input image according to the image features of sample objects learned in the training process, determines the image region of the target object and outputs it. It can be seen that the above method can only segment the objects whose image features are learned when segmenting the target object in the image. For example, the segmentation model only learns the image features of cats, so the segmentation model only has the ability to segment cats from the image. However, since the objects contained in the image can be various, it will only be able to segment limited objects from the image, resulting in low accuracy of object segmentation in the image. SUMMARY

[0003] The embodiments of the present application provide an image processing method, device, equipment and medium, which can improve the accuracy of object segmentation in an image.

[0004] In one aspect, the embodiments of the present application provide an image processing method, which comprises:

[0005] obtaining a target image containing a target object, and obtaining an image feature vector of the target image;

[0006] obtaining an object query vector used for querying the target object from the target image, and generating an object feature vector associated with the target image based on the object query vector and the image feature vector; the object feature vector is used to represent object query information determined from the target image, and the object query information is used to indicate that the target object in the target image is queried by the object query vector;

[0007] obtaining a class code vector corresponding to each object class in an object class set, and determining the similarity between the object feature vector and the class code vector corresponding to each object class to determine the belonging object class of the target object from the object class set; the class code vector is obtained by text coding a class text associated with the object class.

[0008] In one aspect, the embodiments of the present application provide an image processing method, which comprises:

[0009] obtain a first target image containing a first target object and a second target image containing a second target object, and obtain a first image feature vector of the first target image and a second image feature vector of the second target image;

[0010] obtain an object query vector; the object query vector is used for querying the first target object from the first target image and used for querying the second target object from the second target image;

[0011] generate a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and generate a second object feature vector associated with the second target image based on the object query vector and the second image feature vector; the object feature vector associated with any target image is used for representing object query information determined from the any target image, and the object query information is used for indicating that the target object in the target image is queried by the object query vector;

[0012] determine an associated target object matching the first target object from the second target object based on the first object feature vector and the second object feature vector.

[0013] In one aspect, an embodiment of the present application provides an image processing apparatus, which comprises:

[0014] a first obtaining module, configured to obtain a target image containing a target object, and obtain an image feature vector of the target image;

[0015] a first processing module, configured to obtain an object query vector used for querying the target object from the target image, and generate an object feature vector associated with the target image based on the object query vector and the image feature vector; the object feature vector is used for representing object query information determined from the target image, and the object query information is used for indicating that the target object in the target image is queried by the object query vector;

[0016] The first processing module is further configured to obtain a class code vector corresponding to each object class in an object class set, and determine a similarity between the object feature vector and the class code vector corresponding to each object class, so as to determine a belonging object class of the target object from the object class set; the class code vector is obtained by text coding of a class text associated with the object class.

[0017] In one aspect, an embodiment of the present application provides an image processing apparatus, which comprises:

[0018] a second obtaining module, configured to obtain a first target image containing a first target object and a second target image containing a second target object, and obtain an image feature vector of the first target image and an image feature vector of the second target image;

[0019] The second processing module is configured to obtain an object query vector, the object query vector being used to query the first target object from the first target image and to query the second target object from the second target image.

[0020] The second processing module is further configured to generate a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and to generate a second object feature vector associated with the second target image based on the object query vector and the second image feature vector. The object feature vector associated with any target image is used to represent object query information determined from the any target image, and the object query information is used to indicate that the target object in the target image is queried by the object query vector.

[0021] The second processing module is further configured to determine, based on the first object feature vector and the second object feature vector, an associated target object matching the first target object from the second target object.

[0022] In one aspect, an electronic device is provided, which includes a processor and a memory. The memory is configured to store a computer program including program instructions. The processor is configured to invoke the program instructions to perform some or all of the steps of the above method.

[0023] In one aspect, a computer readable storage medium is provided, which stores a computer program including program instructions. The program instructions, when executed by a processor, are configured to perform some or all of the steps of the above method.

[0024] Accordingly, according to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions. The computer instructions, when executed by a processor, can implement some or all of the steps of the above method.

[0025] In the embodiments of the present application, a target image containing a target object can be acquired, an image feature vector of the target image is acquired, an object query vector used for querying the target object from the target image is acquired, an object feature vector associated with the target image is generated based on the object query vector and the image feature vector; the object feature vector can represent object query information determined from the target image by the object query vector, and the object query information can indicate that the object query vector queries one object or does not query any object; a class code vector corresponding to each object class in an object class set is acquired, and a similarity between the object feature vector and the class code vector corresponding to each object class is determined to determine a belonging object class of the target object from the object class set; it can be understood that this way can decouple the recognition logic of the object position and the object class of one target object, the target object in the target image can be queried from the target image by the object query vector first, and the corresponding object feature vector is generated, and the object position and the object class are determined by the object feature vector respectively. In addition, the object classes in the object class set can also be adjusted flexibly, such as adding or deleting some object classes, so that the belonging object classes of more kinds of objects in the target image can be recognized, and the accuracy and flexibility of object segmentation in the image can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0027] Figure 1 A schematic diagram of an image processing scene provided by an embodiment of the present application;

[0028] Figure 2 A flowchart of an image processing method provided by an embodiment of the present application;

[0029] Figure 3 A schematic diagram of an object mask provided by an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a scene for determining an object mask provided by an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a scene for highlighting a target image provided by an embodiment of the present application;

[0032] Figure 6 A flowchart of an image processing method provided by an embodiment of the present application;

[0033] Figure 7 A scene diagram of target object segmentation provided by an embodiment of the present application;

[0034] Figure 8 A flow diagram of an image processing method provided by an embodiment of the present application;

[0035] Figure 9 A scene diagram of target object segmentation provided by an embodiment of the present application;

[0036] Figure 10 A structure diagram of an image processing device provided by an embodiment of the present application;

[0037] Figure 11 A structure diagram of an image processing device provided by an embodiment of the present application;

[0038] Figure 12 A structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0040] The image processing method proposed by the embodiments of the present application is implemented in an electronic device, which can be a server or a terminal. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto.

[0041] An image processing scene diagram based on the image processing method can be as shown in Figure 1 Figure 1 ​A network architecture is proposed, which can include a service server and a user terminal cluster, wherein the user terminal cluster can include one or more user terminals, and the number of user terminals in the user terminal cluster is not limited. There can be a communication connection between the user terminals in the user terminal cluster. At the same time, any user terminal in the user terminal cluster can have a communication connection with the service server, so that each user terminal in the user terminal cluster can interact with the service server through the communication connection. The communication connection is not limited to the connection mode, which can be directly or indirectly connected through wired communication, or directly or indirectly connected through wireless communication, or through other means, which is not limited in this application. In addition, it can be understood that the electronic device involved in the embodiments of the present application can be Figure 1 the service server shown in FIG. 1, or Figure 1 any one of the user terminals in the user terminal cluster shown in FIG. 1.

[0042] For example, in the embodiments of the present application, the server can obtain a target image uploaded by any user terminal, and realize object segmentation of the image through the image processing method proposed in the present application. For example, when the server obtains a target image containing a target object, it can obtain an image feature vector of the target image, and obtain an object query vector for querying the target object from the target image, generate an object feature vector associated with the target image based on the object query vector and the image feature vector, generate an object mask image corresponding to the object feature vector based on the object feature vector and the image feature vector, the object mask image can be used to mark the target object in the target image, obtain a class code vector corresponding to each object class in the object class set based on the object mask image, determine the similarity between the object feature vector and the class code vector corresponding to each object class, and determine the object class to which the target object belongs (that is, determine the specific object meaning of the target object segmented from the target image) based on the similarity. It can be understood that the object query vector may or may not query the target object from the target image. Therefore, if the target object in the target image is determined to be queried based on the object feature vector and the image feature vector (which can be understood as the target object segmented from the target image), such as determining the target object to be queried based on the object mask image, the object class set is obtained, and the class code vector corresponding to each object class is obtained. Subsequently, the target object and the object class to which it belongs can be associated and displayed in the target image.

[0043] It can be understood that if the object mask image includes a target mask value (such as 1) for marking a target object, it is determined that the target object is found, and the image region formed by the target mask value is the object position of the target object. If the object mask image does not include a target mask value (such as only 0 in the object mask value) for marking a target object, it is determined that no object is found. The image size of the object mask image is consistent with the image size of the target image.

[0044] It can be understood that the above object category set can include one or more object categories, and it should be understood that the object categories in the object category set can be adjusted, such as adding or deleting object categories, so the technical solution of the present application does not limit the types of objects that can be identified from the image, and the object segmentation efficiency and flexibility of the image can be improved.

[0045] Optionally, in some embodiments, the electronic device can perform the image processing method according to actual business needs to achieve efficient object segmentation. The technical solution of the present application can be applied to any object segmentation scene. For example, in an object movement detection scene, the electronic device can obtain a video to be segmented, extract video frames in the video to be segmented, and sequentially perform object segmentation on each video frame according to the technical solution of the present application, and generate a corresponding object movement trajectory based on the object position and object category of the object identified in each video frame. For another example, in a vehicle recognition scene, the electronic device can obtain a driving image, and perform object segmentation on the driving image according to the technical solution of the present application to determine vehicles, pedestrians, buildings, etc. included in the driving image. The application scenarios are not limited here.

[0046] Optionally, the data involved in the present application, such as the target image, the segmented target object, etc., can be stored in a database or a blockchain, such as a blockchain distributed system, and the present application does not limit it.

[0047] It should be noted that in the specific embodiments of the present application, when the scene involving obtaining user information and related data, such as obtaining a target image uploaded by a user, the user's permission or consent is required. That is, when the embodiments of the present application are applied to specific products or technologies, the collection, use and processing of related user data comply with relevant laws, regulations and standards of countries and regions. For example, a prompt information can be sent through an interactive interface to prompt which data will be collected or obtained, and the type and content of the data can be prompted to the user through a list or the like. Only after receiving a confirmation operation or instruction allowing data collection on the interactive interface, the related data collection and processing will be further performed.

[0048] It can be understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided by the embodiments of the present application. The technical solutions provided by the embodiments of the present application can also be applied to other scenarios. For example, those skilled in the art can know that as the system architecture evolves and new business scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0049] Based on the above description, the embodiments of the present application provide an image processing method, which can be executed by the electronic device mentioned above. Please refer to Figure 2 , Figure 2 The flowchart of the image processing method provided by the embodiments of the present application is shown in the figure.

[0050] As shown in Figure 2 , the flow of the image processing method of the embodiments of the present application can include the following:

[0051] S101, obtaining a target image containing a target object, and obtaining an image feature vector of the target image.

[0052] The target image can be an image uploaded by a user terminal, or can be a video frame extracted from a video, or can be an image downloaded from the network, or can be an image obtained from a business database (such as a database constructed in a vehicle recognition scenario). The source of the target image is not limited here.

[0053] The target image can include a target object to be queried and identified. The target object can also be referred to as an object. There can be one or more target objects in a target image. The number of target objects in the target image is not limited. It can be understood that the image part other than the target object in the target image can be referred to as image background.

[0054] In some embodiments, the image feature vector of the target image can be obtained by dividing the target image into R image blocks, obtaining an embedding vector of each image block, inputting the embedding vector of each image block into the image encoder, sequentially outputting an image feature sub-vector of each image block, and taking the R image feature sub-vectors as the image feature vector. It can be understood that an image feature sub-vector is used to represent the image features of an image region in the target image, that is, to represent the image features of the corresponding image block. Wherein, the division of the target image can be performed in pixel granularity. For example, the target image is divided in 1x1 pixel granularity, and each image block obtained at this time corresponds to a pixel point. Or, the target image is divided in 2x1 pixel granularity, and each image block obtained at this time corresponds to two pixel points. For example, the pixels of the target image are 4x4, when the target image is divided in 1x1 pixel granularity, the R image blocks obtained are 16 image blocks; when the target image is divided in 2x2 pixel granularity, the R image blocks obtained are 4 image blocks. Here, the pixel granularity used for the division of the target image is not limited. That is, R is related to the pixels (i.e. image size) of the target image.

[0055] Wherein, for ease of understanding, subsequent examples are taken as the target image being divided in 1x1 pixel granularity, such as the image size of the target image being HxW, that is, the R image blocks are HW image blocks. The image encoder can be an encoder in a transformer (a kind of neural network model), and when the image encoder performs image processing based on the embedding vector of any image block to output the corresponding image feature sub-vector, it can fuse the embedding vectors of the image blocks associated with the any image block (such as the neighboring image blocks of the any image block, etc.) to make the image feature sub-vector of any image block not only contain image information of any image block, but also include image information of the associated image blocks.

[0056] Wherein, if the image feature sub-vector is a C-dimensional vector, the image feature vector includes R C-dimensional vectors (which can also be represented as HxW C-dimensional vectors). It can be understood that the R C-dimensional vectors are obtained by sequentially arranging the HxW C-dimensional vectors. Alternatively, the image encoder can belong to an image segmentation model, and the embedding vector associated with the pixel value can be pre-set or obtained by training the image segmentation model at the same time.

[0057] S102, obtaining an object query vector for querying a target object from a target image, and generating an object feature vector associated with the target image based on the object query vector and the image feature vector.

[0058] The object query vector is used to query the target object (i.e., the object) contained in the target image. There can be N object query vectors, where N is a positive integer. Each object query vector can query the object contained in the target image. The query process and principle of each object query vector are the same. It can be understood that the object query vector can be trained simultaneously with the image segmentation model. During training, N initial object query vectors can be obtained, and the N initial object query vectors are trained during training of the image segmentation model to obtain the N object query vectors herein. It can be understood that N can be a preset parameter, which is not limited herein.

[0059] In some embodiments, generating the object feature vector associated with the target image can be inputting the object query vector and the image feature vector into an image decoder, and performing feature interaction on the object query vector and the image feature vector by the image decoder to obtain the object feature vector corresponding to the object query vector. The image decoder belongs to the image segmentation model, and the image decoder can be a decoder in a transformer.

[0060] Optionally, one object query vector can correspond to one object feature vector. The object feature vector is used to represent the object query information determined from the target image. The object query information can indicate whether the target object is queried from the image feature vector by the object query vector. It can be understood that one object query vector queries the target object from the target image by performing feature interaction with the image feature vector. When one object query vector fails to query any target object, the object query information represented by the corresponding object feature vector indicates that the target object is not queried, that is, the object query vector may query and identify from the target image is not the target object, but the image background. When one object query vector queries one target object, the object query information represented by the corresponding object feature vector indicates that the target object is queried.

[0061] In the image decoder, the feature interaction on the object query vector and the image feature vector can be as follows:

[0062] Q0=softmax(Q1*F T )F+Q1

[0063] wherein Q1∈R NxC represents the N C-dimensional object query vectors processed by the image decoder; F∈R RxC represents the R C-dimensional image feature sub-vectors processed by the image decoder. softmax(Q1*F T)A similarity matrix of size NxR can be obtained, which is used to measure the similarity between each object query vector and each image feature sub-vector, and the higher the similarity represents that the image block corresponding to the image feature sub-vector is more likely to belong to the object region of the target object queried by the corresponding object query vector, i.e., the image block is more likely to represent the target object. softmax(Q1*F T )F is an NxC vector, which is used to represent the image features of the target image being sequentially assigned to each object query vector according to the similarity matrix. The resulting Q0∈R NxC represents an NxC object feature vector fused by vector addition, and the NxC object query vector corresponds to the NxC object feature vector one by one. The object feature vector can be understood as a new object query vector centered on the object queried by the object query vector.

[0064] That is, the target object contained in the target image can be queried by the object query vector, and the feature information of the queried target object can be extracted from the target image, and the corresponding object feature vector can be generated based on the feature information and the object query vector. When an object query vector queries the target object contained therein, the feature information of the specific target object queried can be extracted by the object query vector, and the corresponding object feature vector is generated. At this time, the object query information represented by the object feature vector indicates that the target object in the target image is queried by the object query vector. When an object query vector does not query the target object contained therein, the object query information represented by the corresponding object feature vector indicates that no target object in the target image is queried by the object query vector, i.e., the queried target object is empty.

[0065] S103, generating an object mask graph corresponding to the object feature vector based on the object feature vector and the image feature vector.

[0066] In some embodiments, generating the object mask graph can specifically be that each image feature sub-vector is respectively convolved by the object feature vector to obtain a convolution result corresponding to each image feature sub-vector, determining a mask value corresponding to an image region indicated by each image feature sub-vector based on the convolution result corresponding to each image feature sub-vector, and generating the object mask graph by the mask value corresponding to the image region indicated by each image feature sub-vector. It can be understood that the image size of the object mask graph is consistent with the image size of the target image. It can be understood that the object mask graph can be used to mark the target object. The object mask graph can include a target mask value (such as 1) and a non-target mask value (such as 0), and the target mask value is used to mark the target object queried by the object query vector, and the non-target mask value is a mask value other than the target mask value.

[0067] In some embodiments, the convolution processing of each image feature sub-vector by the object feature vector can be specifically that the object feature vector is input into a convolution kernel generator (also referred to as a multilayer perceptron) to obtain a dynamic convolution kernel corresponding to the object feature vector, and each image feature sub-vector is convolved by the dynamic convolution kernel to obtain a convolution result corresponding to each image feature sub-vector. The convolution kernel generator can belong to the image segmentation model and is obtained in the training process of the image segmentation model. That is, one object query vector can obtain one object mask image.

[0068] It can be understood that the determination of the mask value corresponding to the image region indicated by each image feature sub-vector based on the convolution result corresponding to each image feature sub-vector can be that if the convolution result corresponding to any image feature sub-vector is greater than or equal to a preset threshold, it is determined that the mask value corresponding to the image region indicated by any image feature sub-vector is a target mask value, that is, the image region indicated by any image feature sub-vector belongs to the region where the target object is located; if the convolution result corresponding to any image feature sub-vector is less than the preset threshold, it is determined that the mask value corresponding to the image region indicated by any image feature sub-vector is a non-target mask value, that is, the image region indicated by any image feature sub-vector does not belong to the region where the target object is located.

[0069] It can be understood that each object feature vector generates an object mask image. The object mask image can represent the position and shape of the target object queried out by the corresponding object query vector in the target image. When an object query vector queries an object, the corresponding output object mask image is composed of 0 and 1, and in the object mask image, the region where the corresponding target object is located is composed of 1, and the remaining image region is composed of 0. When an object query vector does not query a target object, the corresponding output object mask image is composed of 0. It can be understood that each object query vector can query one target object from the target image, and the target objects queried by different object query vectors can be different. It can be understood that when the target image is divided by 1x1 pixel granularity, one mask value in the obtained object mask image corresponds to one pixel. When the target image is divided by 2x2 pixel granularity, one mask value in the obtained object mask image corresponds to four pixels.

[0070] For example, the target image includes an object A, and it is determined that the object A is queried from the target image by an object query vector, and the corresponding generated object mask image can be as shown in Figure 3 Figure 3 ​A schematic diagram of an object mask map provided by an embodiment of the present application; wherein the object mask map records a mask value for a corresponding pixel point in the target image, if any pixel point (i.e. the image region indicated by the image feature sub-vector) belongs to the region where the object A is located, the mask value recorded in the object mask map for the any pixel point is a target mask value (i.e. 1), if any pixel point does not belong to the region where the object A is located, the mask value recorded in the object mask map for the any pixel point is a non-target mask value (i.e. 0).

[0071] For example, as shown in Figure 4 Figure 4 A scene schematic diagram for determining an object mask map provided by an embodiment of the present application; wherein the image feature vector of the target image includes R C-dimensional image feature vectors 40 (such as V1 C , V2 C ,..., V R C ), the object query vector includes N C-dimensional object query vectors 41 (such as U1 C , U2 C ,..., U N C ), each object query vector is sequentially multiplied by each image feature sub-vector and then output through a softmax function to obtain an NxR similarity matrix 42 (S 11 , S 12 ,..., S RN ), and the object feature vector 43 (such as W1 C , W2 C ,..., W N C ) is determined by the similarity matrix 42, the image feature vector 40 and the object query vector 41; taking the object query vector U2 C as an example, the object query vector U2 C is sequentially multiplied by the image feature sub-vector V1 C , V2 C ,..., V R C and then output through a softmax function to obtain the similarity (S C , S C ,..., S C ) between the object query vector U2 R and the image feature vector V1 C , V2 21 ,..., V 22 , and the similarity (S 2N , S 21 ,..., S 22 ) is output through a softmax function.2N ) and the corresponding image feature vector (V1 C 、V2 C ,...,V R C ) after vector multiplication, and then with the object query vector U2 C Perform vector addition to obtain the object query vector U2 C The corresponding object feature vector W2 C .

[0072] For example, based on the above object query vector U2 C The corresponding object feature vector W2 C The object mask image 44 may be generated by converting the object feature vector W2 C The convolution kernel generator 45 is input to obtain the dynamic convolution kernel 46 (a C-dimensional vector), and the image feature vector 40 (V1 C 、V2 C ,...,V R C ) is convolved to obtain the image feature vector 40 (V1 C 、V2 C ,...,V R C )The corresponding convolution result 47(Q1 C 、Q2 C ,...,Q R C ), based on the convolution result 47(Q1 C 、Q2 C ,...,Q R C ) Determine the corresponding image feature vector 40 (V1 C 、V2 C ,...,V R C ) indicates the image area (i.e., pixel point) corresponding to the mask value 48, that is, the mask value corresponding to each pixel point in the target image can be obtained, and the object mask map 44 is generated based on the mask value 48 corresponding to each pixel point.

[0073] S104. Based on the object mask map, obtain the category encoding vector corresponding to each object category in the object category set, and determine the similarity between the object feature vector and the category encoding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set.

[0074] In the object mask map, if the target mask value for marking the target object is included, the class code vector corresponding to each object class in the object class set is obtained; if the target mask value for marking the target object is not included in the object mask map, the class code vector corresponding to each object class in the object class set is not obtained.

[0075] It can be understood that if the object mask map includes the target mask value for marking the target object, it is determined that the object query information indicates that the target object is queried out through the object query vector. Alternatively, if the object mask map includes the target mask value and the number of target mask values is greater than or equal to the preset number, it is determined that the object query information indicates that the target object is queried out through the object query vector. Correspondingly, if the object mask map does not include the target mask value for marking the target object, it is determined that the object query information indicates that the target object is not queried out through the object query vector. Alternatively, if the object mask map includes the target mask value and the number of target mask values is less than the preset number, it is determined that the object query information indicates that the target object is not queried out through the object query vector. That is, one object query feature obtains one object feature vector, and one object feature vector determines one object mask map. Step S104 is performed for each object mask map.

[0076] In some embodiments, the object class set can include a plurality of preset object classes, and the number of object classes can be preset and is not limited herein. For example, the object classes are vehicle, airplane, cat, dog, etc.; each object class has an associated class text, for example, the object class is vehicle, and the class text is vehicle; the class text associated with the object class can be subjected to text encoding processing (such as using an encoder in a transformer model or any text encoder), to obtain a text encoding vector corresponding to the class text associated with the object class, which can be used as the class code vector corresponding to the object class. One object class corresponds to one class code vector. It can be understood that the object classes in the object class set can be configured arbitrarily, which can improve the flexibility of class recognition of objects contained in the image.

[0077] In the object mask map, if the target mask value for marking the target object is included, the class code vector corresponding to each object class in the object class set is obtained; if the target mask value for marking the target object is not included in the object mask map, the class code vector corresponding to each object class in the object class set is not obtained; if the target mask value for marking the target object is included, the class code vector corresponding to each object class in the object class set is obtained; if the target mask value for marking the target object is not included in the object mask map, the class code vector corresponding to each object class in the object class set is not obtained.

[0078] It can be understood that the object class to which the target object belongs can be determined from the object class set based on the similarity, for example, the object class with the highest similarity in the object class set is determined as the object class to which the target object belongs. Subsequently, the target object and the object class to which it belongs can be associated and displayed in the target image.

[0079] In some embodiments, the associatively displaying the target object and the object category to which the target object belongs can be determining an object region in the target image indicated by the object mask graph, and associatively displaying the target object and the object category to which the target object belongs in the object region. It can be understood that the target image is masked based on the target mask value in the object mask graph, and the mask region in the masked target image where the target mask value is located is taken as the object region indicated by the object mask graph in the target image. It can be understood that the object region indicated by the object mask graph is the object region of the identified target object in the target image.

[0080] For example, the associatively displaying the target object and the object category to which the target object belongs in the object region can be highlighting the target object in the object region and generating a category label carrying the object category to which the target object belongs. For example, the image data in the object region is subjected to tone adjustment processing to obtain an adjusted object region, and the adjusted object region is superimposed on the target image to obtain a superimposed target image containing the highlighted target object, and a category label corresponding to the object category to which the target object belongs is generated for the highlighted target object in the superimposed target image. It can be understood that when multiple objects are queried from the target object by using multiple object query vectors, that is, multiple object mask graphs are generated, an adjusted object region can be obtained based on each object mask graph in the above manner, and the multiple adjusted object regions can be superimposed in the target image, so that multiple segmented objects can be viewed in the target image.

[0081] For example, as shown in FIG. 5, Figure 5 As shown in FIG. 5, Figure 5 is a scene diagram provided by an embodiment of the present application for highlighting a target image; the target image 51 is subjected to mask processing based on the object mask graph 50 to obtain a mask region indicated by the object mask graph 50, which can be taken as an object region 53 in which the object 52 in the target image is located. The image data in the object region 53 is subjected to tone adjustment processing to obtain an adjusted object region 54, and the adjusted object region 54 is superimposed on the target image 51. Meanwhile, the target image 51 is subjected to mask processing based on the object mask graph 55 to obtain a mask region indicated by the object mask graph 55, which can be taken as an object region 57 in which the object 56 in the target image is located. The image data in the object region 57 is subjected to tone adjustment processing to obtain an adjusted object region 58, and the adjusted object region 58 is superimposed on the target image 51 to obtain a superimposed target image 59, and a category label corresponding to the object category to which the highlighted object 52 and the object 56 belong is generated in the superimposed target image 59.

[0082] By the above manner, the object segmentation in the image can be realized, the target object contained in the target image can be queried out through the object query vector (at this time, the physical meaning of the target object is not determined), and the object feature vector containing the feature information of the target object is output, the object mask graph corresponding to the queried target object can be generated through the object feature vector, the object region of the target object in the target image is determined through the object mask graph, and the object category to which the target object belongs can be determined through the object feature vector. It can be understood that this manner is to respectively query the object region of one or more target objects contained in the target image from the target image, and then determine the object category to which the one or more target objects belong, through the logical decoupling manner, the multiple objects contained in the target image can be queried, and the actual object category of the queried object can be determined based on the object category set. The efficiency and flexibility of object segmentation can be improved, and the robustness and use value of the image segmentation model can be improved.

[0083] In the embodiment of the application, the recognition logic of the object position and the object category of an object can be decoupled, the target object in the target image can be queried out through the object query vector, and the corresponding object feature vector can be generated, and the object position and the object category can be determined through the object feature vector. In addition, the object categories in the object category set can also be flexibly adjusted, such as adding or deleting some object categories, so that the object categories of more objects in the target image can be recognized, and the accuracy and flexibility of object segmentation in the image can be improved.

[0084] Please refer to Figure 6 , Figure 6 A flowchart of an image processing method provided by the embodiment of the application is shown in the figure, and the method can be executed by the electronic device mentioned above. As shown in Figure 6 The flow of the image processing method in the embodiment of the application can include the following steps:

[0085] S201, a target image containing a target object is acquired, and an image feature vector of the target image is acquired. The specific implementation of step S201 can be referred to the related description of the above embodiment, which will not be repeated here.

[0086] S202, an object query vector for querying the target object from the target image is acquired, and an object feature vector associated with the target image is generated based on the object query vector and the image feature vector.

[0087] The image feature vector includes R image feature sub-vectors, and one image feature sub-vector is used to represent the image feature of one image region in the target image. Therefore, the object feature vector associated with the target object can be determined as follows: determining the region probability that the image region indicated by each image feature sub-vector belongs to an object region, based on the object query vector and each image feature sub-vector, respectively, the object region being used to represent the region of the target object in the target image; and performing fusion processing on the R image feature sub-vectors and the object query vector based on the region probability indicated by each image feature sub-vector, to obtain the object feature vector. The object feature vector is used to represent the object query information determined from the target image. It can be understood that the region probability that the image region indicated by any image feature sub-vector belongs to the object region is the similarity between the object query vector and any image feature sub-vector.

[0088] In some embodiments, the fusion processing on the R image feature sub-vectors and the object query vector to obtain the object feature vector can be specifically performed as follows: determining the to-be-fused feature vector corresponding to each image feature sub-vector from each image feature sub-vector based on the region probability indicated by each image feature sub-vector, respectively; and performing fusion processing on the to-be-fused feature vector corresponding to each image feature sub-vector and the object query vector to obtain the object feature vector. It can be understood that the to-be-fused feature vector corresponding to any image feature sub-vector can be a product vector obtained by performing vector multiplication on the region probability indicated by any image feature sub-vector and any image feature sub-vector. In addition, the sum of the to-be-fused feature vector corresponding to each image feature sub-vector and the object query vector can be used as the object feature vector.

[0089] S203, generating an object mask corresponding to the object feature vector based on the object feature vector and the image feature vector.

[0090] S204, if the target mask value used to mark the target object is included in the object mask, obtaining a class encoding vector corresponding to each object class in the object class set, and determining the similarity between the object feature vector and the class encoding vector corresponding to each object class.

[0091] S205, determining the object class to which the target object belongs from the object class set based on the similarity. The specific implementation of steps S203-S205 can be referred to the related description of the above embodiments, which will not be repeated here.

[0092] S206, highlighting the target object in the target image in a preset display manner, and displaying the highlighted target object and the object class to which the target object belongs in association.

[0093] In some embodiments, highlighting the target object in the target image can be, based on the target mask value in the object mask map, performing mask processing on the target image, taking the mask region in the target image where the target mask value is located as the object region of the target object in the target image, and highlighting the target object on the object region in the target image in a preset display manner.

[0094] In some embodiments, highlighting the target object in the target image can be, based on the target mask value in the object mask map, performing mask processing on the target image, taking the mask region in the target image where the target mask value is located as the object region of the target object in the target image, and highlighting the target object on the object region in the target image in a preset display manner.

[0095] For example, as shown in FIG. 7, Figure 7 Figure 7 is a scene schematic diagram of target object segmentation provided by an embodiment of the present application; wherein the object segmentation process of the image can be: obtaining a target image 70, inputting the target image 70 into an image encoder 71 to obtain an image feature vector 72 (HxW image feature sub-vectors) of the target image 70, obtaining N object query vectors 73, and inputting the N object query vectors 73 and the image feature vector 72 into an image decoder 74 to obtain an object feature vector 75 corresponding to each object query vector 73; generating an object mask map 76 corresponding to each object feature vector 75 through a convolution kernel generator based on the object feature vector 75 corresponding to each object query vector 73 and the image feature vector 72; if the object feature vector 1 associated with the target image 70 is determined from the image feature vector 72 based on the object query vector 1, and the object mask map Figure 1 does not contain the target mask value (for example, all mask values are 0), indicating that the object query vector 1 fails to query any object from the target image 70; or, if the object feature vector 2 associated with the target image 70 is determined from the image feature vector 72 based on the object query vector 2, and the object mask map Figure 2 ​If the target mask value is included, it indicates that the object query vector 2 queries the target object 77 from the target image 70, and therefore, the object feature vector 75 of any object (such as the reference object feature vector 78) can be selected from the N object feature vectors 75 based on the object mask graph 76 corresponding to each object feature vector 75. The object class set 79 is obtained, and the class text 710 associated with each object class in the object class set 79 (such as a person, a vehicle, a cat, a dog, etc.) is input into the text encoder to obtain the class encoding vector 711 corresponding to each object class. The similarity 712 between each reference object feature vector 78 and the class encoding vector 771 corresponding to each object class is determined, and the object class 713 to which the target object 77 indicated by each reference object feature vector 78 belongs is determined based on the similarity 712. The target object 77 and the object class 713 to which it belongs can be displayed in association in the target image 70. For example, the target image 70 is masked based on the target mask value in the object mask graph 76 corresponding to the reference object feature vector 78 to obtain the object region where the target object 77 is located, and the target object 77 in the object region is subjected to tone adjustment processing. The adjusted object region is superimposed on the target image 70 to obtain the target object 77 highlighted, and the target object 77 highlighted and the object class 713 to which it belongs are displayed in association in the target image 70.

[0096] The object query vector, the image encoder, the image decoder, and the convolution kernel generator mentioned above can be obtained by model training of the image segmentation model. The training process can be as follows: obtaining a training sample image and a preset object class set, the training sample image carrying an object mask label and an object class label, generating N initial object query vectors, and using a text encoder to encode the class text associated with each object class in the object class set to obtain a class encoding vector corresponding to each object class. The training sample image is input into the image encoder in the image segmentation model to obtain an image feature vector, the image feature vector and the N initial object query vectors are input into the image decoder, and the object feature vector corresponding to the N initial object query vectors is output by the image decoder. The object feature vector is input into the convolution kernel generator to obtain a dynamic convolution kernel, and the object mask graph is generated by the dynamic convolution kernel and the object feature vector. The object class corresponding to each object class is determined based on the class encoding vector corresponding to each object class and the object feature vector. The N initial object query vectors are trained based on the object mask graph, the corresponding object class, the object mask label, and the object class label, and the image encoder, the image decoder, and the convolution kernel generator in the image segmentation model, so as to obtain the trained image segmentation model and the N object query vectors.

[0097] Therefore, the object segmentation of the image segmentation model guided by the text encoder can query the location and shape of the contained object from the target image through the object query vector, and determine the object category to which the object queried by the object query vector belongs in combination with the category encoding vector, so as to realize the object recognition and segmentation in the image. That is, the present scheme first proposes to use the category encoding vector obtained by the text encoder to guide the classification and segmentation of objects of any category, solving the problem that the traditional method can only classify a fixed category set. It can be understood that the preset object categories in the object category set can be updated according to different scenes. For example, the object category set used in the training stage can only include 10 object categories, but in the application stage, new object categories can be added to the object category set, for example, 20 object categories can be included at this time, so that the above-mentioned method can expand the object categories that can be recognized, and has the ability to recognize new categories of objects, thereby improving the flexibility and accuracy of object segmentation.

[0098] In the embodiments of the present application, the object position and object category recognition logic of an object can be decoupled, the target object in the target image can be queried from the target image through the object query vector, and the corresponding object feature vector can be generated, and the object position and object category can be determined through the object feature vector. In addition, the object categories in the object category set can also be flexibly adjusted, such as adding or deleting some object categories, so that the object categories of more kinds of objects in the target image can be recognized, and the accuracy and flexibility of object segmentation in the image can be improved.

[0099] Please refer to Figure 8 , Figure 8 A flowchart of an image processing method provided by the embodiments of the present application is shown in the figure. The method can be executed by the electronic device mentioned above. As shown in Figure 8 The flow of the image processing method in the embodiments of the present application can include the following steps:

[0100] S301, obtaining a first target image containing a first target object and a second target image containing a second target object, and obtaining a first image feature vector of the first target image and a second image feature vector of the second target image.

[0101] The first target object in the first target image can be one or more. The second target object in the second target image can be one or more. For example, the first target image and the second target image can be video frames cut from a video. The object movement trajectory can be generated based on the objects segmented from each video frame. Here, two adjacent video frames are taken as an example, that is, the first target image and the second target image.

[0102] S302, acquire an object query vector.

[0103] S303, generate a first object feature vector associated with a first target image based on the object query vector and a first image feature vector, and generate a second object feature vector associated with a second target image based on the object query vector and a second image feature vector.

[0104] S304, generate a first object mask map corresponding to the first object feature vector based on the first object feature vector and the first image feature vector, and generate a second object mask map corresponding to the second object feature vector based on the second object feature vector and the second image feature vector. The specific implementation of steps S302-S304 can be referred to the above embodiment description.

[0105] S305, determine an associated target object matching the first target object from the second target object based on the first object feature vector and the second object feature vector, and generate a moving track of the associated target object based on the first object mask map and the second object mask map.

[0106] The first object feature vector associated with the first target image includes a first object feature vector corresponding to a first target object, and the second object feature vector associated with the second target image includes a second object feature vector corresponding to a second target object. It can be understood that the first object feature associated with the first target image includes a first object feature vector recognized by the query and a first object feature vector recognized by the query. Therefore, the first object feature vector recognized by the query can be determined from the first object feature vector associated with the first target image, that is, whether the first object feature vector recognizes the first target object in the first target image can be determined by the first object mask corresponding to the first object feature vector. It can be understood that when a first object feature vector is determined to recognize a first target object in the first target image, it means that the first object feature vector is a first object feature vector corresponding to a first target object. For example, the first object feature vector associated with the first target image includes U first object feature vectors corresponding to U first target objects, and the second object feature vector associated with the second target image includes V second object feature vectors corresponding to V second target objects; U and V are positive integers. Correspondingly, each first target object corresponds to a first object mask, and each second target object corresponds to a second object mask. That is, for example, the first target object recognized by the object query vector from the first target image is U, and the second target object recognized by the object query vector from the second target image is V. U and V are positive integers. For example, the object query vector is N, corresponding to N first object feature vectors, and corresponding to N first object masks. From the N first object masks, it is determined that U first object masks recognize the first target object from the first target image, so the first object feature vectors corresponding to the U first object masks are regarded as the first object feature vectors corresponding to the U first target objects.

[0107] In some embodiments, determining the associated target object can be based on the first object feature vectors corresponding to the U first target objects and the second object feature vectors corresponding to the V second target objects, determining an object correlation between each first target object and each second target object, respectively; determining a second target object matching any first target object from the V second target objects based on the object correlation, and taking the determined second target object as the associated target object. The vector similarity between the object feature vector corresponding to any first object and the object feature vector corresponding to any second object can be taken as the object correlation between any first object and any second object. It can be understood that one object query vector obtains one first object feature vector, one first object feature vector corresponds to one first object mask graph, when the first object mask graph indicates that the first target object is identified by querying the first target image, the first object feature vector corresponding to the first object mask graph is determined as the first object feature vector corresponding to the first target object.

[0108] The determination of the second target object matching any first target object from the V second target objects based on the object correlation can be determining the second target object matching each first target object from the V second target objects, and determining the second target object matching any first target object according to the second target object matching each first target object based on the Hungarian algorithm. The second target object matching any first target object refers to the second target object with the maximum object correlation with any first target object from the V second target objects. It can be understood that if a first target object has a matching second target object, it means that the first target object and the matching second target object are the same target object, i.e., the first target object moves from the location in the first target image to the location in the second target image. In other words, the moving track of each target object in the video can be obtained based on the position information of the same target object segmented from any two adjacent video frames.

[0109] In some embodiments, generating the moving track of the associated target object based on the first object mask graph and the second object mask graph can be determining the first position information of the first target object matching the associated target object in the first target image based on the first object mask graph, determining the second position information of the associated target object in the second target image based on the second object mask graph, and generating the moving track of the associated target object through the first position information and the second position information. The first object mask graph corresponding to the first object feature vector includes the first object mask graphs corresponding to the U first target objects, and the second object mask graph corresponding to the second object feature vector includes the second object mask graphs corresponding to the U second target objects.

[0110] That is, the position information indicated by the first object mask corresponding to the first target object matched with the associated target object in the first target image is determined as the position information of the associated target object in the first target image, and the position information indicated by the second object mask corresponding to the associated target object in the second target image is determined as the position information of the associated target object in the second target image; the movement track of the associated target object is generated through the position information of the associated target object in the first target image and the position information of the associated target object in the second target image. It can be understood that the object region of the first target object matched with the associated target object in the first target image can be determined through the first object mask corresponding to the first target object matched with the associated target object, so as to obtain the position information in the first target image. The object region of the second target object matched with the associated target object in the second target image can be determined through the second object mask corresponding to the associated target object, so as to obtain the position information in the second target image. Thus, the movement track of the associated target object in the first target image and the second target image can be generated. Meanwhile, when the movement track is generated, the object category to which the associated target object belongs can also be determined through the above-mentioned manner, and the associated target object is displayed in association in the first target image and the second target image.

[0111] For example, as Figure 9 , Figure 9A scene schematic diagram of target object segmentation provided for an embodiment of the present application; wherein a first target image and a second target image are acquired, a first image feature vector of the first target image and a second image feature vector of the second target image are acquired; N object query features are acquired; N first object feature vectors associated with the first target image are generated based on the N object query vectors and the first image feature vector, and N second object feature vectors associated with the second target image are generated based on the N object query vectors and the second image feature vector, N first object mask graphs corresponding to the N first object feature vectors are generated based on the N first object feature vectors and the first image feature vector, and N second object mask graphs corresponding to the N second object feature vectors are generated based on the N second object feature vectors and the second image feature vector, U first object mask graphs that recognize the first target object are acquired from the N first object mask graphs, and the U first object mask graphs are taken as the first object feature vectors corresponding to the U first target objects (1, 2,..., U), V second object mask graphs that recognize the second target object are acquired from the N second object mask graphs, and the V second object mask graphs are taken as the second object feature vectors corresponding to the V second target objects (1, 2,..., V); that is, U first target objects are segmented from the first target image and V second target objects are segmented from the second target image through the N object query vectors; the object correlation matrix between the U first target objects and the V second target objects can be constructed based on the similarity between the first object feature vector corresponding to each first target object and the second object feature vector corresponding to each second target object (wherein, Figure 9 The object correlation matrix shown in the object correlation matrix indicates the object correlation between the first target object 1 and the second target object 2, and the object correlation between the first target object and the second target object; the rest is the same), and through the Hungarian algorithm, the second target object matched with any first target object is determined based on the object correlation matrix, such as the first target object 1 matched with the second target object 2, and the first target object 2 matched with the second target object 3, so the position information of the first target object 1 in the first target image can be determined based on the first object mask graph corresponding to the first target object 1, and the position information of the second target object 2 in the second target image can be determined based on the second object mask graph corresponding to the second target object 2, to generate the moving track of the second target object 2, and the position information of the first target object 2 in the first target image can be determined based on the first object mask graph corresponding to the first target object 2, and the position information of the second target object 3 in the second target image can be determined based on the second object mask graph corresponding to the second target object 3, to generate the moving track of the second target object 3.

[0112] In the embodiments of the present application, the object position and object category recognition logic of an object can be decoupled, the target object in the target image can be queried out from the target image through the object query vector, and the corresponding object feature vector can be generated, and the object position and the same target object can be determined through the object feature vector, so as to generate the moving track of the target object in the first target image and the second target image.

[0113] Please refer to Figure 10 , Figure 10 The structure diagram of an image processing device provided in the present application is shown. It should be noted that Figure 10 The image processing device shown in the figure is used to execute the method of the embodiments of the present application Figure 2 and Figure 6 For the convenience of description, only the parts related to the embodiments of the present application are shown, and the specific technical details are not disclosed, and the embodiments shown in the present application Figure 2 and Figure 6 The image processing device 1000 can include a first acquisition module 1001, a first processing module 1002. Wherein:

[0114] The first acquisition module 1001 is used to acquire a target image containing a target object, and acquire an image feature vector of the target image;

[0115] The first processing module 1002 is used to acquire an object query vector used to query the target object from the target image, and generate an object feature vector associated with the target image based on the object query vector and the image feature vector;

[0116] The first processing module 1002 is also used to generate an object mask image corresponding to the object feature vector based on the object feature vector and the image feature vector; the object mask image is used to mark the target object;

[0117] The first processing module 1002 is also used to acquire a class code vector corresponding to each object category in an object category set based on the object mask image, and determine the similarity between the object feature vector and the class code vector corresponding to each object category, so as to determine the belonging object category of the target object from the object category set.

[0118] In some embodiments, the image feature vector includes R image feature sub-vectors, and one image feature sub-vector is used to represent the image feature of one image region in the target image;

[0119] When the first processing module 1002 is used to generate the object feature vector associated with the target image based on the object query vector and the image feature vector, it is specifically used to:

[0120] determine a region probability that an image region indicated by each image feature sub-vector belongs to the object region based on the object query vector and each image feature sub-vector respectively, wherein the object region is used to represent a region of the target object in the target image;

[0121] fuse the R image feature sub-vectors and the object query vector based on the region probability indicated by each image feature sub-vector to obtain an object feature vector.

[0122] In some embodiments, the first processing module 1002, when used to fuse the R image feature sub-vectors and the object query vector based on the region probability indicated by each image feature sub-vector to obtain an object feature vector, is specifically configured to:

[0123] determine a to-be-fused feature vector corresponding to each image feature sub-vector from each image feature sub-vector based on the region probability indicated by each image feature sub-vector respectively;

[0124] fuse the to-be-fused feature vector corresponding to each image feature sub-vector and the object query vector to obtain the object feature vector.

[0125] In some embodiments, the image feature vector includes R image feature sub-vectors, and one image feature sub-vector is used to represent image features of one image region in the target image.

[0126] The first processing module 1002, when used to generate an object mask graph corresponding to the object feature vector based on the object feature vector and the image feature vector, is specifically configured to:

[0127] perform convolution processing on each image feature sub-vector based on the object feature vector to obtain a convolution result corresponding to each image feature sub-vector;

[0128] determine a mask value corresponding to an image region indicated by each image feature sub-vector based on the convolution result corresponding to each image feature sub-vector;

[0129] generate the object mask graph based on the mask value corresponding to the image region indicated by each image feature sub-vector.

[0130] In some embodiments, the first processing module 1002, when used to obtain a class encoding vector corresponding to each object class in the object class set based on the object mask graph, is specifically configured to:

[0131] if the object mask graph includes a target mask value used to mark the target object, obtain the class encoding vector corresponding to each object class in the object class set;

[0132] If the object mask map does not include a target mask value for marking the target object, the category encoding vector corresponding to each object category in the object category set is not obtained.

[0133] In some embodiments, the first processing module 1002 is further configured to:

[0134] Based on the target mask value in the object mask image, mask processing is performed on the target image, and the mask area where the target mask value is located in the target image after mask processing is used as the object area of ​​the target object in the target image;

[0135] The target object in the object area is highlighted in the target image according to a preset display mode, and the highlighted target object and the object category to which it belongs are displayed in association.

[0136] The specific implementation of the first acquisition module and the first processing module can be found in the description of the above embodiment, which will not be described in detail here. It should be understood that the description of the beneficial effects obtained by adopting the same method will not be described in detail either.

[0137] See Figure 11 , Figure 11 This is a schematic diagram of the structure of an image processing device provided by this application. It should be noted that, Figure 11 The image processing device shown is used to execute the present application Figure 8 For the sake of convenience, only the part related to the embodiment of the present application is shown, and the specific technical details are not disclosed. Figure 8 The embodiment shown. The image processing device 1100 may include: a second acquisition module 1101, a second processing module 1102. Wherein:

[0138] A second acquisition module 1101 is configured to acquire a first target image containing a first target object and a second target image containing a second target object, and acquire an image feature vector of the first target image and an image feature vector of the second target image;

[0139] The second processing module 1102 is configured to obtain an object query vector; the object query vector is used to query a first target object from the first target image and to query a second target object from the second target image;

[0140] The second processing module 1102 is further configured to generate a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and to generate a second object feature vector associated with the second target image based on the object query vector and the second image feature vector;

[0141] The second processing module 1102 is further configured to generate a first object mask corresponding to the first object feature vector based on the first object feature vector and the first image feature vector, and generate a second object mask corresponding to the second object feature vector based on the second object feature vector and the second image feature vector; the first object mask is used to mark the first target object, and the second object mask is used to mark the second target object.

[0142] The second processing module 1102 is further configured to determine, based on the first object feature vector and the second object feature vector, an associated target object matching the first target object from the second target object, and generate a movement track of the associated target object based on the first object mask and the second object mask.

[0143] The first object feature vector associated with the first target image includes U first object feature vectors corresponding to U first target objects, and the second object feature vector associated with the second target image includes V second object feature vectors corresponding to V second target objects; U and V are positive integers.

[0144] When the second processing module 1102 is configured to determine, based on the first object feature vector and the second object feature vector, an associated target object matching the first target object from the second target object, the second processing module 1102 is specifically configured to:

[0145] Determine an object correlation between each first target object and each second target object based on the U first object feature vectors corresponding to the U first target objects and the V second object feature vectors corresponding to the V second target objects.

[0146] Determine, based on the object correlation, a second target object matching any first target object of the U first target objects from the V second target objects, and take the determined second target object as the associated target object.

[0147] When the second processing module 1102 is configured to generate a movement track of the associated target object based on the first object mask and the second object mask, the second processing module 1102 is specifically configured to:

[0148] Determine first position information of the first target object matching the associated target object in the first target image based on the first object mask, and determine second position information of the associated target object in the second target image based on the second object mask.

[0149] Generate the movement track of the associated target object based on the first position information and the second position information.

[0150] The specific implementation of the second obtaining module and the second processing module can refer to the description of the above embodiments, which will not be described here. It should be understood that the beneficial effects obtained by using the same method will not be described here.

[0151] Please refer to Figure 12 , Figure 12 A structural schematic diagram of an electronic device is provided in the embodiments of the present application. As shown in the figure, Figure 12 The electronic device 1200 includes at least one processor 1201, a memory 1202. Optionally, the electronic device can also include a network interface. Among them, the processor 1201, the memory 1202 and the network interface can interact with data, the network interface is controlled by the processor 1201 for transmitting and receiving messages, the memory 1202 is used for storing computer programs, the computer programs include program instructions, and the processor 1201 is used for executing the program instructions stored in the memory 1202. Among them, the processor 1201 is configured to call the program instructions to execute the above method.

[0152] The memory 1202 can include a volatile memory, such as a random-access memory (RAM); the memory 1202 can also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the memory 1202 can also include a combination of the above kinds of memories.

[0153] The processor 1201 can be a central processing unit (CPU). In an embodiment, the processor 1201 can also be a graphics processing unit (GPU). The processor 1201 can also be a combination of CPU and GPU. The processor 1201 can be used to call the device control application stored in the memory 1202 to execute the above Figure 2 、 Figure 6 and Figure 8 corresponding to the description of the image processing method in the embodiments, the description of the image processing device in the foregoing Figure 10 and Figure 11 corresponding embodiments will not be described here. In addition, the description of the beneficial effects of using the same method will not be described here.

[0154] In a specific implementation, the apparatus, processor, memory and the like described in the embodiments of the present application can perform the implementation manner described in the above method embodiment, and can also perform the implementation manner described in the embodiments of the present application, which will not be described here.

[0155] The embodiment of the present application further provides a computer (readable) storage medium, the computer storage medium stores a computer program, the computer program includes program instructions, and the program instructions are executed by a processor to enable the processor to execute part or all of the steps performed in the above method embodiments. Optionally, the computer storage medium can be volatile or non-volatile. The computer readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, and the like; and the data storage area can store data created according to the use of the blockchain node and the like.

[0156] The embodiment of the present application provides a computer program product, which can include a computer program, and the computer program can implement part or all of the steps in the above method when executed by a processor, which will not be described here.

[0157] In the present document, "multiple" refers to two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0158] A person of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer storage medium, which can be a computer readable storage medium. The program can include the processes of the above method embodiments when executed.

[0159] The above only discloses some embodiments of the present application, and of course cannot limit the scope of the rights of the present application. A person of ordinary skill in the art can understand that the above embodiments can be implemented in whole or in part, and equivalent changes made according to the claims of the present application still fall within the scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a target image containing a target object, and acquire an image feature vector of the target image; Obtaining an object query vector for searching the target object from the target image, and generating an object feature vector associated with the target image based on the object query vector and the image feature vector; the object feature vector is used to represent object query information determined from the target image, and the object query information is used to indicate the target object in the target image queried using the object query vector; Obtain a category coding vector corresponding to each object category in an object category set, and determine the similarity between the object feature vector and the category coding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set; the category coding vector is obtained by text encoding the category text associated with the object category.

2. The method according to claim 1, characterized in that The image feature vector includes R image feature sub-vectors, and one image feature sub-vector is used to represent the image features of an image region in the target image; The generating the object feature vector associated with the target image based on the object query vector and the image feature vector includes: Determining, based on the object query vector and each image feature sub-vector, a region probability that the image region indicated by each image feature sub-vector belongs to the object region; The object region is used to represent a region of the target object in the target image; The R image feature sub-vectors and the object query vector are fused based on the region probability indicated by each image feature sub-vector to obtain the object feature vector.

3. The method according to claim 2, characterized in that The fusing the R image feature sub-vectors and the object query vector based on the region probability indicated by each image feature sub-vector to obtain the object feature vector includes: Determining the to-be-fused feature vector corresponding to each image feature sub-vector from each image feature sub-vector based on the region probability indicated by each image feature sub-vector; The feature vector to be fused corresponding to each image feature sub-vector and the object query vector are fused to obtain the object feature vector.

4. The method according to claim 1, wherein The method further comprises: generating an object mask map corresponding to the object feature vector based on the object feature vector and the image feature vector; the object mask map is used to mark the target object; Based on the object mask map, determine whether to execute the step of obtaining the category encoding vector corresponding to each object category in the object category set, and determine the similarity between the object feature vector and the category encoding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set.

5. The method according to claim 4, characterized in that The image feature vector includes R image feature sub-vectors, and one image feature sub-vector is used to represent the image features of an image region in the target image; Generating an object mask map corresponding to the object feature vector based on the object feature vector and the image feature vector includes: Performing convolution processing on each image feature sub-vector using the object feature vector to obtain a convolution result corresponding to each image feature sub-vector; Determining a mask value corresponding to the image region indicated by each image feature sub-vector based on a convolution result corresponding to each image feature sub-vector; The object mask map is generated by using the mask value corresponding to the image area indicated by each image feature sub-vector.

6. The method according to claim 4, characterized in that The step of determining whether to execute the step of obtaining the category encoding vector corresponding to each object category in the object category set based on the object mask map, and determining the similarity between the object feature vector and the category encoding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set, includes: If the object mask map includes a target mask value for marking the target object, obtaining a category encoding vector corresponding to each object category in the object category set, and determining a similarity between the object feature vector and the category encoding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set; If the object mask map does not include a target mask value for marking the target object, the category encoding vector corresponding to each object category in the object category set is not obtained.

7. The method according to claim 6, characterized in that The method further comprises: performing mask processing on the target image based on the target mask value in the object mask map, and using a mask region in the target image after the mask processing where the target mask value is located as an object region of the target object in the target image; The target object in the object area is highlighted in the target image according to a preset display mode, and the highlighted target object and the object category to which it belongs are displayed in association.

8. An image processing method, characterized in that: The method comprises: Acquire a first target image containing a first target object and a second target image containing a second target object, and acquire a first image feature vector of the first target image and a second image feature vector of the second target image; Obtaining an object query vector; the object query vector is used to query the first target object from the first target image and to query the second target object from the second target image; generating a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and generating a second object feature vector associated with the second target image based on the object query vector and the second image feature vector; the object feature vector associated with any target image is used to represent object query information determined from the any target image, the object query information being used to indicate a target object in the target image queried using the object query vector; An associated target object that matches the first target object is determined from the second target object based on the first object feature vector and the second object feature vector.

9. The method according to claim 8, characterized in that The first object feature vector associated with the first target image includes U first object feature vectors corresponding to the first target objects, and the second object feature vector associated with the second target image includes V second object feature vectors corresponding to the second target objects; U and V are both positive integers; The determining, from the second target object based on the first object feature vector and the second object feature vector, an associated target object that matches the first target object includes: Determining an object correlation between each first target object and each second target object based on the first object feature vectors corresponding to the U first target objects and the second object feature vectors corresponding to the V second target objects; A second target object matching any one of the U first target objects is determined from the V second target objects based on the object relevance, and the determined second target object is used as the associated target object.

10. The method according to claim 8, characterized in that The method further comprises: generating a first object mask map corresponding to the first object feature vector based on the first object feature vector and the first image feature vector, and generating a second object mask map corresponding to the second object feature vector based on the second object feature vector and the second image feature vector; the first object mask map is used to mark the first target object, and the second object mask map is used to mark the second target object; A movement trajectory of the associated target object is generated based on the first object mask map and the second object mask map.

11. The method according to claim 10, characterized in that The generating the movement trajectory of the associated target object based on the first object mask map and the second object mask map includes: determining first position information of a first target object matching the associated target object in the first target image based on the first object mask map, and determining second position information of the associated target object in the second target image based on the second object mask map; A movement trajectory of the associated target object is generated using the first position information and the second position information.

12. An image processing device, characterized in that: The device comprises: A first acquisition module is used to acquire a target image containing a target object and obtain an image feature vector of the target image; a first processing module, configured to obtain an object query vector for searching the target object from the target image, and generate an object feature vector associated with the target image based on the object query vector and the image feature vector; the object feature vector is configured to represent object query information determined from the target image, and the object query information is configured to indicate the target object in the target image queried using the object query vector; The first processing module is further used to obtain a category coding vector corresponding to each object category in the object category set, and determine the similarity between the object feature vector and the category coding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set; the category coding vector is obtained by text encoding the category text associated with the object category.

13. An image processing device, characterized in that: The device comprises: A second acquisition module is configured to acquire a first target image containing a first target object and a second target image containing a second target object, and acquire an image feature vector of the first target image and an image feature vector of the second target image; A second processing module is configured to obtain an object query vector; the object query vector is used to query the first target object from the first target image and to query the second target object from the second target image; The second processing module is further configured to generate a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and to generate a second object feature vector associated with the second target image based on the object query vector and the second image feature vector; the object feature vector associated with any target image is used to represent object query information determined from the any target image, the object query information being used to indicate a target object in the target image queried using the object query vector; The second processing module is further configured to determine, from the second target object, an associated target object that matches the first target object based on the first object feature vector and the second object feature vector.

14. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 11.

Images