Image processing method, apparatus, device, and medium

Abstract

Embodiments of the present application disclose an image processing method, device and equipment and medium, which are applied to the technical field of image processing. The method comprises: obtaining a target image containing a target object, an image feature vector and an object query vector; generating an object feature vector based on the object query vector and the image feature vector; obtaining a class code vector corresponding to each object class in an object class set; determining the similarity between the object feature vector and the class code vector corresponding to each object class, so as to determine the belonging object class of the target object from the object class set. The embodiments of the present application can improve the accuracy of object segmentation in an image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and medium. Background Technology

[0002] Currently, object segmentation (also known as image segmentation) is an important task in the field of image processing. "Segmentation" refers to identifying target objects (such as vehicles, airplanes, etc.) from an image and determining their location within the image. Existing object segmentation methods typically involve inputting an image into a pre-trained segmentation model. The model then uses the image features of sample objects learned during training to detect and identify the corresponding target objects in the input image, determining the image region of the target object and outputting it. It is evident that these methods can only segment objects that have learned the corresponding image features. For example, if the segmentation model has only learned the image features of a cat, it can only segment cats from an image. However, since images can contain a wide variety of objects, this limits the number of objects that can be segmented, resulting in low accuracy in object segmentation. Summary of the Invention

[0003] This application provides an image processing method, apparatus, device, and medium that can improve the accuracy of object segmentation in images.

[0004] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0005] Obtain the target image containing the target object, and obtain the image feature vector of the target image;

[0006] Obtain an object query vector for querying a target object from a target image, and generate an object feature vector associated with the target image based on the object query vector and the image feature vector; the object feature vector is used to characterize the object query information determined from the target image, and the object query information is used to indicate how to query the target object in the target image through the object query vector;

[0007] The category encoding vector corresponding to each object category in the object category set is obtained, and the similarity between the object feature vector and the category encoding vector corresponding to each object category is determined, so as to determine the object category to which the target object belongs from the object category set; the category encoding vector is obtained by text encoding the category text associated with the object category.

[0008] On one hand, embodiments of this application provide an image processing method, which includes:

[0009] Obtain a first target image containing a first target object and a second target image containing a second target object, and obtain a first image feature vector of the first target image and a second image feature vector of the second target image;

[0010] Obtain the object query vector; the object query vector is used to query the first target object from the first target image and to query the second target object from the second target image;

[0011] A first object feature vector associated with a first target image is generated based on an object query vector and a first image feature vector, and a second object feature vector associated with a second target image is generated based on an object query vector and a second image feature vector; the object feature vector associated with any target image is used to characterize the object query information determined from the any target image, and the object query information is used to indicate that the target object in the target image is retrieved through the object query vector;

[0012] Based on the feature vectors of the first and second objects, the associated target objects that match the first target object are determined from the second target object.

[0013] On one hand, embodiments of this application provide an image processing apparatus, which includes:

[0014] The first acquisition module is used to acquire a target image containing the target object and to acquire the image feature vector of the target image;

[0015] The first processing module is used to obtain an object query vector for querying a target object from a target image, and to generate an object feature vector associated with the target image based on the object query vector and the image feature vector; the object feature vector is used to characterize the object query information determined from the target image, and the object query information is used to indicate how to query the target object in the target image through the object query vector;

[0016] The first processing module is further configured to obtain the category encoding vector corresponding to each object category in the object category set, and determine the similarity between the object feature vector and the category encoding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set; the category encoding vector is obtained by text encoding the category text associated with the object category.

[0017] On one hand, embodiments of this application provide an image processing apparatus, which includes:

[0018] The second acquisition module is used to acquire a first target image containing a first target object and a second target image containing a second target object, and to acquire the image feature vector of the first target image and the image feature vector of the second target image;

[0019] The second processing module is used to obtain object query vectors; the object query vectors are used to query the first target object from the first target image and to query the second target object from the second target image.

[0020] The second processing module is further configured to generate a first object feature vector associated with a first target image based on an object query vector and a first image feature vector, and to generate a second object feature vector associated with a second target image based on an object query vector and a second image feature vector; the object feature vector associated with any target image is used to characterize the object query information determined from the any target image, and the object query information is used to indicate that the target object in the target image is queried through the object query vector;

[0021] The second processing module is also used to determine the associated target object that matches the first target object from the second target object based on the first object feature vector and the second object feature vector.

[0022] On one hand, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute some or all of the steps in the above method.

[0023] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform some or all of the steps in the above-described method.

[0024] Accordingly, according to one aspect of this application, a computer program product or computer program is provided, which includes computer instructions that, when executed by a processor, can implement some or all of the steps in the above-described method.

[0025] In this embodiment, a target image containing the target object can be acquired, along with an image feature vector. An object query vector for retrieving the target object from the target image is then obtained. Based on the object query vector and the image feature vector, an object feature vector associated with the target image is generated. This object feature vector represents the object query information determined from the target image through the object query vector, indicating whether the object query vector retrieves an object or not. A category encoding vector corresponding to each object category in the object category set is acquired, and the similarity between the object feature vector and the category encoding vector corresponding to each object category is determined to identify the object category of the target object from the object category set. This method decouples the identification logic of the object location and object category of a target object. The target object in the target image can be retrieved first through the object query vector, and a corresponding object feature vector can be generated. The object location and object category are then determined using this object feature vector. Furthermore, the object categories in the object category set can be flexibly adjusted, such as by adding or deleting some object categories, thereby enabling the identification of multiple object categories in the target image and improving the accuracy and flexibility of object segmentation in the image. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of an image processing scenario provided in an embodiment of this application;

[0028] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0029] Figure 3 A schematic diagram of an object mask provided in an embodiment of this application;

[0030] Figure 4 A schematic diagram of a scenario for determining an object mask image, provided in an embodiment of this application;

[0031] Figure 5 This application provides a schematic diagram of a scene for highlighting a target image, as illustrated in an embodiment of the present application.

[0032] Figure 6 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0033] Figure 7 A schematic diagram of a target object segmentation scenario provided for an embodiment of this application;

[0034] Figure 8 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0035] Figure 9 A schematic diagram of a target object segmentation scenario provided for an embodiment of this application;

[0036] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0037] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0038] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0040] The image processing method proposed in this application is implemented in an electronic device, which can be a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these.

[0041] A schematic diagram of an image processing scenario proposed based on this image processing method can be shown as follows: Figure 1 As shown, Figure 1A network architecture is proposed, which may include a service server and a user terminal cluster. The user terminal cluster may include one or more user terminals; the number of user terminals in the cluster is not limited. Communication connections can exist between the user terminals in the cluster. Simultaneously, any user terminal in the cluster can have a communication connection with the service server, enabling each user terminal in the cluster to interact with the service server via this connection. The communication connection method is not limited; it can be established directly or indirectly via wired communication, wireless communication, or other methods. This application does not impose any limitations on this method. Furthermore, it is understood that the electronic devices involved in the embodiments of this application may be… Figure 1 The business server shown can also be Figure 1 Any user terminal in the user terminal cluster shown.

[0042] For example, in this embodiment, the server can acquire a target image uploaded by any user terminal and perform object segmentation of the image using the image processing method proposed in this application. For instance, when acquiring a target image containing a target object, the server can acquire the image feature vector of the target image and an object query vector for retrieving the target object from the target image. Based on the object query vector and the image feature vector, an object feature vector associated with the target image is generated. Based on the object feature vector and the image feature vector, an object mask map corresponding to the object feature vector is generated. The object mask map can be used to mark the target object in the target image. Based on the object mask map, the category encoding vector corresponding to each object category in the object category set is obtained. The similarity between the object feature vector and the category encoding vector corresponding to each object category is determined. Based on the similarity, the object category to which the target object belongs is determined from the object category set (that is, the specific object meaning of the target object segmented from the target image is determined). It can be understood that the object query vector may or may not retrieve the target object from the target image. Therefore, if the target object in the target image is determined by querying the object query vector based on the object feature vector and image feature vector (this can be understood as segmenting the target object from the target image), for example, by generating an object mask map based on the object feature vector and image feature vector, and determining the target object through the object mask map, then a set of object categories is obtained, and the category encoding vector corresponding to each object category is obtained. Subsequently, the target object and its corresponding object category can be associated and displayed in the target image.

[0043] It can be understood that if the object mask image includes a target mask value (e.g., 1) to mark the target object, it indicates that the target object has been found, and the image region formed by the target mask value is the object location. If the object mask image does not include a target mask value to mark the target object (e.g., the object mask value only contains 0), it indicates that no object has been found. The image size of the object mask image is consistent with the image size of the target image.

[0044] It is understood that the above object category set may include one or more object categories. It should be understood that the object categories in the object category set can be adjusted, such as adding or deleting object categories. Therefore, the technical solution of this application does not limit the types of objects that can be identified from the image, which can improve the efficiency and flexibility of image object segmentation.

[0045] Optionally, in some embodiments, the electronic device can execute the image processing method to achieve efficient object segmentation according to actual business needs. The technical solution of this application can be applied to any object segmentation scenario. For example, in an object movement detection scenario, the electronic device can acquire a video to be segmented, extract video frames from the video to be segmented, and sequentially perform object segmentation on each video frame according to the technical solution of this application, generating a corresponding object movement trajectory based on the object position and object category identified in each video frame. As another example, in a vehicle recognition scenario, the electronic device can acquire a driving image and perform object segmentation on the driving image according to the technical solution of this application to determine the vehicles, pedestrians, buildings, etc., included in the driving image. The application scenarios are not limited here.

[0046] Optionally, the data involved in this application, such as target images and segmented target objects, can be stored in a database or in a blockchain. This application does not limit the storage of such data through a blockchain distributed system.

[0047] It should be noted that in specific embodiments of this application, when scenarios involving the acquisition of user information and related data, such as acquiring target images uploaded by users, user permission or consent is required. That is, when the embodiments of this application are applied to specific products or technologies, the collection, use, and processing of relevant user data comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, prompts can be issued through an interactive interface to indicate what data will be collected or acquired. Specifically, the types and content of this data can be presented to the user through lists or other means. Further data collection and processing will only proceed after a confirmation or instruction to allow data collection is received on the interactive interface.

[0048] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0049] Based on the foregoing description, this application proposes an image processing method that can be executed by the aforementioned electronic device. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application.

[0050] like Figure 2 As shown, the flow of the image processing method in this application embodiment may include the following:

[0051] S101. Obtain the target image containing the target object, and obtain the image feature vector of the target image.

[0052] The target image can be an image uploaded by a user terminal, a video frame extracted from a video, an image downloaded from the internet, or an image obtained from a business database (such as a database built for vehicle recognition scenarios). The source of the target image is not limited here.

[0053] The target image may include the target object to be identified. A target object is also called an object. A target image may contain one or more target objects. There is no limit to the number of target objects in a target image. It can be understood that the portion of the target image other than the target objects can be referred to as the image background.

[0054] In some embodiments, obtaining the image feature vector of the target image can be achieved by dividing the target image into R image blocks, obtaining the embedding vector of each image block, inputting the embedding vector of each image block into an image encoder, sequentially outputting the image feature sub-vectors of each image block, and using the R image feature sub-vectors as the image feature vector. It can be understood that an image feature sub-vector is used to characterize the image features of an image region in the target image, that is, to characterize the image features of the corresponding image block. The division of the target image can be done at the pixel level. For example, dividing the target image at a 1x1 pixel level results in each image block corresponding to one pixel. Alternatively, dividing the target image at a 2x1 pixel level results in each image block corresponding to two pixels. For example, if the target image has 4x4 pixels, dividing it at a 1x1 pixel level results in 16 image blocks (R images); dividing it at a 2x2 pixel level results in 4 image blocks (R images). Here, the pixel level used for dividing the target image is not limited. That is, R is related to the number of pixels in the target image (i.e., the image size).

[0055] For ease of understanding, we will use a 1x1 pixel granularity to divide the target image as an example. For instance, if the target image size is HxW, that is, R image blocks, or HW image blocks. The image encoder can be an encoder in a transformer (a type of neural network model). When processing the image based on the embedding vector of any image block to output the corresponding image feature sub-vector, the image encoder can fuse the embedding vectors of image blocks associated with that image block (such as neighboring image blocks). This ensures that the image feature sub-vector of any image block not only contains the image information of that image block itself but also the image information of associated image blocks.

[0056] Wherein, if the image feature vector is a C-dimensional vector, then the image feature vector includes R C-dimensional vectors (or H x W C-dimensional vectors). It can be understood that the R C-dimensional vectors are obtained by sequentially arranging the H x W C-dimensional vectors. Optionally, the image encoder can belong to the image segmentation model, and the pixel value-related embedding vectors can be pre-defined or obtained during simultaneous training with the image segmentation model.

[0057] S102. Obtain the object query vector used to query the target object from the target image, and generate the object feature vector associated with the target image based on the object query vector and the image feature vector.

[0058] The object query vector is used to retrieve target objects (i.e., objects) from the target image. There can be N object query vectors, where N is a positive integer. Each object query vector can retrieve objects from the target image. The query process and principle are the same for each object query vector. It can be understood that the object query vectors can be obtained simultaneously with the image segmentation model. During training, N initial object query vectors can be obtained, and these N initial object query vectors are used to train the image segmentation model, resulting in the N object query vectors used here. It can be understood that N can be a preset parameter and is not limited here.

[0059] In some embodiments, generating the object feature vector associated with the target image can be achieved by inputting the object query vector and the image feature vector into an image decoder, whereby the image decoder performs feature interaction on the object query vector and the image feature vector to obtain the object feature vector corresponding to the object query vector. The image decoder belongs to the image segmentation model and can be a decoder within a transformer.

[0060] Optionally, an object query vector can correspond to an object feature vector. This object feature vector represents the object query information determined from the target image. This object query information indicates whether the object query vector retrieves the target object from the image feature vector. It can be understood that an object query vector retrieves the target object from the target image through feature interaction with the image feature vector. When an object query vector fails to retrieve any target object, the object query information represented by the corresponding object feature vector indicates that no target object was retrieved; that is, the object query vector may have retrieved and identified the image background instead of the target object. When an object query vector retrieves a target object, the object query information represented by the corresponding object feature vector indicates that the target object was retrieved.

[0061] The feature interaction between the object query vector and the image feature vector in the image decoder can be performed as follows:

[0062] Q0 = softmax(Q1 * F T )F+Q1

[0063] Where Q1∈R NxC This represents N C-dimensional object query vectors processed by the image decoder; F∈R RxC This represents R C-dimensional image feature vectors after processing by the image decoder. softmax(Q1*F) TThis yields a similarity matrix of size N x R, used to measure the similarity between each object query vector and each image feature vector. A higher similarity indicates that the image patch corresponding to that feature vector is more likely to belong to the object region of the target object queried by the corresponding object query vector; that is, the image patch is more likely to represent the target object. softmax(Q1*F) T F is a set of N C-dimensional vectors, representing the assignment of image features of the target image to each object query vector based on the similarity matrix. The resulting Q0∈R NxC This represents the N C-dimensional object feature vectors resulting from the addition and fusion of vectors. These N C-dimensional object query vectors correspond one-to-one with the N C-dimensional object feature vectors. The object feature vector can be understood as a new object query vector centered on the object retrieved by the object query vector.

[0064] In other words, object query vectors can be used to query for target objects contained in a target image, extract feature information of the queried target objects from the target image, and generate corresponding object feature vectors based on this feature information and the object query vector. When an object query vector finds a contained target object, the feature information of the specific queried target object can be extracted from the object query vector, and a corresponding object feature vector can be generated. At this time, the object feature vector represents the object query information indicating that the target object in the target image was found through the object query vector. When an object query vector does not find a contained target object, the object query information represented by the corresponding generated object feature vector indicates that no target object was found in the target image through the object query vector, that is, the target object is empty.

[0065] S103. Generate an object mask image corresponding to the object feature vector based on the object feature vector and the image feature vector.

[0066] In some embodiments, generating an object mask map specifically involves performing convolution processing on each image feature vector using the object feature vector to obtain the convolution result corresponding to each image feature vector. Based on the convolution result corresponding to each image feature vector, a mask value corresponding to the image region indicated by each image feature vector is determined. An object mask map is then generated using the mask value corresponding to the image region indicated by each image feature vector. It is understood that the image size of the object mask map is consistent with the image size of the target image. It is understood that the object mask map can be used to mark target objects. The object mask map can include target mask values ​​(e.g., 1) and non-target mask values ​​(e.g., 0). The target mask value is used to mark the target object retrieved by the object query vector, and the non-target mask value is a mask value other than the target mask value.

[0067] In some embodiments, convolution processing is performed on each image feature vector using the object feature vector to obtain the convolution result corresponding to each image feature vector. Specifically, the object feature vector can be input into a convolution kernel generator (also known as a multilayer perceptron) to obtain a dynamic convolution kernel corresponding to the object feature vector. This dynamic convolution kernel is then used to convolve each image feature vector to obtain the convolution result corresponding to each image feature vector. The convolution kernel generator can be part of the image segmentation model and is obtained during the image segmentation model training process. In other words, an object query vector can yield an object mask image.

[0068] It can be understood that determining the mask value corresponding to the image region indicated by each image feature vector based on the convolution result corresponding to each image feature vector can be as follows: if the convolution result corresponding to any image feature vector is greater than or equal to a preset threshold, then the mask value corresponding to the image region indicated by any image feature vector is determined to be the target mask value, that is, the image region indicated by any image feature vector belongs to the region where the target object is located; if the convolution result corresponding to any image feature vector is less than the preset threshold, then the mask value corresponding to the image region indicated by any image feature vector is determined to be the non-target mask value, that is, the image region indicated by any image feature vector does not belong to the region where the target object is located.

[0069] It can be understood that each object feature vector generates an object mask. This object mask represents the position and shape of the target object retrieved by the corresponding object query vector in the target image. When an object query vector retrieves an object, the corresponding output object mask consists of 0s and 1s, with the area representing the target object represented by 1s and the remaining image areas represented by 0s. When an object query vector does not retrieve a target object, the corresponding output object mask consists of 0s. It can be understood that each object query vector can retrieve one target object from the target image, and different object query vectors can retrieve different target objects. Specifically, when the target image is divided into 1x1 pixel granularities, one mask value in the resulting object mask corresponds to one pixel. When the target image is divided into 2x2 pixel granularities, one mask value in the resulting object mask corresponds to four pixels.

[0070] For example, if the target image includes object A, and it is determined that object A can be retrieved from the target image using an object query vector, then the corresponding generated object mask image can be as follows: Figure 3 As shown, Figure 3This is a schematic diagram of an object mask map provided in an embodiment of this application; wherein, the object mask map records a mask value for a corresponding pixel in the target image. If any pixel (i.e., the image region indicated by the image feature vector) belongs to the region where object A is located, the mask value recorded in the object mask map for that pixel is the target mask value (i.e., 1). If any pixel does not belong to the region where object A is located, the mask value recorded in the object mask map for that pixel is the non-target mask value (i.e., 0).

[0071] For example, such as Figure 4 As shown, Figure 4 This application provides a schematic diagram of a scenario for determining an object mask image according to an embodiment of the present application; wherein, the image feature vector of the target image includes R C-dimensional image feature vectors 40 (e.g., represented as V1). C V2 C ... V R C The object query vector includes N C-dimensional object query vectors 41 (e.g., represented as U1). C U2 C ..., U N C For each object query vector, the vector is multiplied sequentially with each image feature subvector, and then the result is passed through the softmax function to output an NxR similarity matrix. 11 S 12 ... S RN The object feature vector 43 (e.g., represented as W1) is determined using the similarity matrix 42, the image feature vector 40, and the object query vector 41. C W2 C ... W N C ); using object query vector U2 C For example, object query vector U2 C sequentially with image feature vector V1 C V2 C ... V R C Perform vector multiplication and then pass the softmax function to output the object lookup vector U2. C With image feature vector V1 C V2 C ... V R C Similarity between (S) 21 S 22 ... S 2N ), and the similarity (S 21 S 22 ... S2N ) and the corresponding image feature vector (V1) C V2 C ... V R C After performing vector multiplication, it is then sequentially multiplied with the object query vector U2. C Perform vector addition to obtain the object query vector U2. C The corresponding object feature vector W2 C .

[0072] For example, based on the above object query vector U2 C The corresponding object feature vector W2 C The object mask diagram 44 can be generated by taking the object feature vector W2. C Input convolution kernel generator 45 to obtain dynamic convolution kernel 46 (a C-dimensional vector), and pass the dynamic convolution kernel 46 sequentially to process image feature vector 40 (V1). C V2 C ... V R C The image feature vector 40(V1) is obtained by convolution processing. C V2 C ... V R C The corresponding convolution result is 47(Q1). C Q2 C ... Q R C Based on this convolution result 47(Q1) C Q2 C ... Q R C Determine the corresponding image feature vector 40 (V1) C V2 C ... V R C The mask value 48 corresponding to the image region (i.e. pixel) indicated by the image region (i.e. pixel) can be obtained, which means that the mask value corresponding to each pixel in the target image can be obtained, and the object mask image 44 is generated based on the mask value 48 corresponding to each pixel.

[0073] S104. Based on the object mask image, obtain the category encoding vector corresponding to each object category in the object category set, and determine the similarity between the object feature vector and the category encoding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set.

[0074] Specifically, based on the object mask image, obtaining the category encoding vector corresponding to each object category in the object category set can be done as follows: if the object mask image includes a target mask value used to mark the target object, then the category encoding vector corresponding to each object category in the object category set is obtained; if the object mask image does not include a target mask value used to mark the target object, then the category encoding vector corresponding to each object category in the object category set is not obtained.

[0075] It is understood that if the object mask image includes target mask values ​​used to mark the target object, then the object query information indicates that the target object was retrieved through the object query vector. Alternatively, if the object mask image includes target mask values ​​and the number of target mask values ​​is greater than or equal to a preset number, then the object query information indicates that the target object was retrieved through the object query vector. Correspondingly, if the object mask image does not include target mask values ​​used to mark the target object, then the object query information indicates that the target object was not retrieved through the object query vector. Alternatively, if the object mask image includes target mask values ​​and the number of target mask values ​​is less than a preset number, then the object query information indicates that the target object was not retrieved through the object query vector. In other words, one object query feature yields one object feature vector, and one object feature vector determines one object mask image. Step S104 is executed for each object mask image.

[0076] In some embodiments, the object category set may include multiple preset object categories, the number of which can be preset and is not limited here. For example, object categories may be vehicles, airplanes, cats, dogs, etc.; each object category has associated category text, for example, if the object category is vehicles, then the category text is vehicles; the category text associated with the object category can be text encoded (e.g., using an encoder in a transformer model or any text encoder) to obtain the text encoding vector corresponding to the category text associated with the object category, and this text encoding vector can be used as the category encoding vector corresponding to the object category. One object category corresponds to one category encoding vector. It can be understood that the object categories in the object category set can be arbitrarily configured, which can improve the flexibility of category recognition of objects contained in an image.

[0077] Specifically, the object feature vector can be multiplied by the category encoding vector corresponding to each object category, and the result of the vector multiplication can be used as the similarity between the object feature vector and the category encoding vector corresponding to each object category.

[0078] This is understandable. The object category of a target object can be determined from a set of object categories based on similarity; for example, the object category with the highest similarity in the set can be identified as the target object's category. Subsequently, the target object and its category can be associated and displayed in the target image.

[0079] In some embodiments, associating and displaying a target object and its associated object category can involve determining the object region indicated by the object mask image in the target image, and then associating and displaying the target object and its associated object category within that object region. It can be understood that the target image is masked based on the target mask value in the object mask image, and the masked region containing the target mask value in the masked target image is taken as the object region indicated by the object mask image in the target image. It can be understood that the object region indicated by the object mask image is the object region of the identified target object in the target image.

[0080] For example, associating a target object with its category within an object region can involve highlighting the target object and generating a category label carrying its category. For instance, adjusting the color tone of image data within the object region to obtain an adjusted object region can be overlaid on the target image to create an overlaid target image containing the highlighted target object. In this overlaid target image, a category label corresponding to the object category of the highlighted target object can be generated. The color tone adjustment process can involve adjusting the color values ​​of the image data within the object region, such as rendering all image data colors as purple, gray, etc. It can be understood that when multiple objects are retrieved from a target object using multiple object query vectors—that is, when generating multiple object mask images—an adjusted object region can be obtained based on each object mask image, as described above. These multiple adjusted object regions can then be overlaid on the target image, allowing multiple segmented objects to be viewed within the target image.

[0081] For example, such as Figure 5 As shown, Figure 5 This is a schematic diagram of a scene for highlighting a target image provided in an embodiment of this application. The target image 51 is masked based on an object mask map 50 to obtain a masked area indicated by the object mask map 50. This masked area can be used as the object region 53 where object 52 is located in the target image. The image data in object region 53 is subjected to tone adjustment processing to obtain an adjusted object region 54, and the adjusted object region 54 is superimposed on the target image 51. Simultaneously, the target image 51 is masked based on an object mask map 55 to obtain a masked area indicated by the object mask map 55. This masked area can be used as the object region 57 where object 56 is located in the target image. The image data in object region 57 is subjected to tone adjustment processing to obtain an adjusted object region 58, and the adjusted object region 58 is superimposed on the target image 51 to obtain a superimposed target image 59. Category labels corresponding to the object categories of the highlighted objects 52 and 56 are generated in the superimposed target image 59.

[0082] The above method enables object segmentation in images. It retrieves target objects from the target image using object query vectors (the physical meaning of these objects is not yet determined), outputting object feature vectors containing their feature information. An object mask map corresponding to the retrieved target objects is generated from these feature vectors. This object mask map determines the object region of the target object in the target image, and the object category of the target object is determined from the object feature vectors. Essentially, this method first retrieves the object regions of one or more target objects from the target image, then determines the object category of each of these objects. Through logical decoupling, it can retrieve multiple objects from the target image and determine the actual object category of each retrieved object based on a set of object categories. This improves the efficiency and flexibility of object segmentation, while also enhancing the robustness and usability of the image segmentation model.

[0083] In this embodiment, the identification logic of an object's location and object category can be decoupled. First, the target object in the target image can be retrieved using an object query vector, and a corresponding object feature vector is generated. The object location and object category are then determined using this object feature vector. Furthermore, the object categories in the object category set can be flexibly adjusted, such as by adding or deleting categories, thereby enabling the identification of a wider range of object categories in the target image. This improves the accuracy and flexibility of object segmentation in images.

[0084] Please see Figure 6 , Figure 6 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. This method can be executed by the aforementioned electronic device. Figure 6 As shown, the flow of the image processing method in this embodiment may include the following:

[0085] S201. Obtain the target image containing the target object, and obtain the image feature vector of the target image. The specific implementation of step S201 can be found in the relevant description of the above embodiments, and will not be repeated here.

[0086] S202. Obtain the object query vector used to query the target object from the target image, and generate the object feature vector associated with the target image based on the object query vector and the image feature vector.

[0087] The image feature vector comprises R image feature sub-vectors, each representing the image features of a region in the target image. Therefore, determining the object feature vector associated with the target object can be achieved by: determining the probability that the image region indicated by each image feature sub-vector belongs to the object region, based on the object query vector and each image feature sub-vector. This object region represents the region of the target object in the target image. The R image feature sub-vectors and the object query vector are then fused based on the region probabilities indicated by each image feature sub-vector to obtain the object feature vector. The object feature vector represents the object query information determined from the target image. It can be understood that the probability that the image region indicated by any image feature sub-vector belongs to the object region is the similarity between the object query vector and any image feature sub-vector.

[0088] In some embodiments, fusing R image feature sub-vectors and an object query vector to obtain an object feature vector can specifically involve determining the corresponding feature vector to be fused from each image feature sub-vector based on the region probability indicated by each image feature sub-vector, and fusing the corresponding feature vector to be fused and the object query vector to obtain the object feature vector. It can be understood that determining the corresponding feature vector to be fused from any image feature sub-vector can be a product vector obtained by multiplying the region probability indicated by any image feature sub-vector with any image feature sub-vector. Furthermore, the sum of the corresponding feature vector to be fused and the object query vector can be used as the object feature vector.

[0089] S203. Generate an object mask image corresponding to the object feature vector based on the object feature vector and the image feature vector.

[0090] S204. If the object mask image includes target mask values ​​for marking target objects, then obtain the category encoding vector corresponding to each object category in the object category set, and determine the similarity between the object feature vector and the category encoding vector corresponding to each object category.

[0091] S205. Determine the object category of the target object from the object category set based on similarity. The specific implementation methods for steps S203-S205 can be found in the relevant descriptions of the above embodiments, and will not be repeated here.

[0092] S206. Highlight the target object in the target image according to the preset display method, and associate the highlighted target object with its category.

[0093] In some embodiments, highlighting a target object in a target image may involve masking the target image based on the target mask value in the object mask image, using the masked area containing the target mask value in the masked target image as the object area of ​​the target object in the target image, and highlighting the target object in the object area in the target image according to a preset display method.

[0094] Highlighting a target object in a target image according to a preset display method can be achieved by adjusting the color tone of the target object in the target image to obtain an adjusted target area, and then overlaying the adjusted target area onto the target image to obtain a target image containing the highlighted target object. Optionally, the specific method for associating the highlighted target object with its corresponding object category can be found in the relevant description of the above embodiments.

[0095] For example, such as Figure 7 As shown, Figure 7 This is a schematic diagram of a target object segmentation scenario provided in an embodiment of this application. The object segmentation process may include: acquiring a target image 70 and inputting it into an image encoder 71 to obtain an image feature vector 72 (H x W image feature sub-vectors); acquiring N object query vectors 73 and inputting the N object query vectors 73 and the image feature vectors 72 into an image decoder 74 to obtain an object feature vector 75 corresponding to each object query vector 73; generating an object mask image 76 corresponding to each object feature vector 75 based on the object feature vector 75 corresponding to each object query vector 73 and the image feature vector 72 using a convolution kernel generator; if an object feature vector 1 associated with the target image 70 is determined from the image feature vector 72 based on the object query vector 1, and an object mask is generated based on the object feature vector 1 and the image feature vector 72... Figure 1 If the target mask value is not included (e.g., all mask values ​​are 0), it means that object query vector 1 failed to find any object in target image 70; or, if object feature vector 2 associated with target image 70 is determined from image feature vector 72 based on object query vector 2, and object mask is generated based on object feature vector 2 and image feature vector 72. Figure 2If the target mask value is included, it means that the object query vector 2 retrieves the target object 77 from the target image 70. Therefore, based on the object mask image 76 corresponding to each object feature vector 75, the object feature vector 75 of any object can be selected from N object feature vectors 75 (for example, represented as the reference object feature vector 78). Obtain the object category set 79, and input the category text 710 associated with each object category in the object category set 79 (for example, people, vehicles, cats, dogs, etc.) into the text encoder to obtain the category encoding vector 711 corresponding to each object category. Determine the similarity 712 between each reference object feature vector 78 and the category encoding vector 771 corresponding to each object category. Based on the similarity 712, the object category 713 of the target object 77 indicated by each reference object feature vector 78 is determined. The queried target object 77 and its object category 713 can be displayed together in the target image 70. For example, the target image 70 is masked based on the target mask value in the object mask image 76 corresponding to the reference object feature vector 78 to obtain the object region where the target object 77 is located. The target object 77 in the object region is then color-adjusted. The adjusted object region is superimposed on the target image 70 to obtain the highlighted target object 77. The highlighted target object 77 and its object category 713 are then displayed together in the target image 70.

[0096] The object query vector, image encoder, image decoder, and convolution kernel generator mentioned above can be obtained by training the image segmentation model. The training process can be as follows: Acquire training sample images and a preset set of object categories. Each training sample image carries an object mask label and its corresponding object category label. Generate N initial object query vectors. Use a text encoder to encode the category text associated with each object category in the object category set, obtaining a category encoding vector for each object category. Input the training sample images into the image encoder of the image segmentation model to obtain image feature vectors. Input the image feature vectors and the N initial object query vectors into the image decoder. The image decoder outputs object feature vectors corresponding to the N initial object query vectors. Input the object feature vectors into a convolution kernel generator to obtain dynamic convolution kernels. Generate an object mask using the dynamic convolution kernels and object feature vectors. Determine the object category based on the category encoding vector and object feature vector corresponding to each object category. Train the N initial object query vectors based on the object mask, the corresponding object category, the object mask label, and the object category label, along with the image encoder, image decoder, and convolution kernel generator in the image segmentation model. This results in a trained image segmentation model and N object query vectors.

[0097] Therefore, by combining an image segmentation model guided by a text encoder, object segmentation can be achieved. Specifically, the location and shape of objects within a target image can be retrieved using object query vectors, and the object category of the retrieved object can be determined by combining the category encoding vector. This enables object recognition and segmentation in the image. In other words, this scheme first proposes using category encoding vectors obtained from a text encoder to guide the classification and segmentation of objects of any category, solving the problem that traditional methods can only classify a fixed set of categories. It can be understood that the preset object categories in this set can be updated according to different scenarios. For example, the object category set used during the training phase may only include 10 object categories, but during the application phase, new object categories can be added to the set, such as including 20 object categories. Therefore, the above method can expand the recognizable object categories, enabling the recognition of new object categories, thereby improving the flexibility and accuracy of object segmentation.

[0098] In this embodiment, the identification logic of an object's location and object category can be decoupled. First, the target object in the target image can be retrieved using an object query vector, and a corresponding object feature vector is generated. The object location and object category are then determined using this object feature vector. Furthermore, the object categories in the object category set can be flexibly adjusted, such as by adding or deleting categories, thereby enabling the identification of a wider range of object categories in the target image. This improves the accuracy and flexibility of object segmentation in images.

[0099] Please see Figure 8 , Figure 8 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. This method can be executed by the aforementioned electronic device. Figure 8 As shown, the flow of the image processing method in this embodiment may include the following:

[0100] S301. Obtain a first target image containing a first target object and a second target image containing a second target object, and obtain a first image feature vector of the first target image and a second image feature vector of the second target image.

[0101] The first target image may contain one or more first target objects. The second target image may contain one or more second target objects. For example, the first and second target images may be video frames extracted from a video. Object movement trajectories can be generated based on the objects segmented from each video frame. Here, we take two adjacent video frames as an example, namely the first and second target images.

[0102] S302, Obtain the object query vector.

[0103] S303. Generate a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and generate a second object feature vector associated with the second target image based on the object query vector and the second image feature vector.

[0104] S304. Generate a first object mask image corresponding to the first object feature vector based on the first object feature vector and the first image feature vector, and generate a second object mask image corresponding to the second object feature vector based on the second object feature vector and the second image feature vector. For specific implementation details of steps S302-S304, please refer to the relevant descriptions in the above embodiments.

[0105] S305. Based on the first object feature vector and the second object feature vector, determine the associated target object that matches the first target object from the second target object, and generate the movement trajectory of the associated target object based on the first object mask map and the second object mask map.

[0106] The first object feature vector associated with the first target image includes the first object feature vector corresponding to the first target object, and the second object feature vector associated with the second target image includes the second object feature vector corresponding to the second target object. It can be understood that the first object feature vector associated with the first target image includes the first object feature vector that identifies the first target object and the first object feature vector that identifies the image background. Therefore, the first object feature vector that identifies the first target object can be determined from the first object feature vector associated with the first target image; that is, it can be determined whether the first object feature vector identifies the first target object in the first target image through the first object mask image corresponding to the first object feature vector. It can be understood that when a first object feature vector is determined to identify a first target object in the first target image, it means that the first object feature vector is the first object feature vector corresponding to a first target object. For example, the first object feature vector associated with the first target image includes the first object feature vectors corresponding to U first target objects, and the second object feature vector associated with the second target image includes the second object feature vectors corresponding to V second target objects; U and V are both positive integers. Correspondingly, each first target object corresponds to a first object mask image, and each second target object corresponds to a second object mask image. In other words, for example, if U first target objects are identified from the first target image using object query vectors, and V second target objects are identified from the second target image using object query vectors, then U and V are both positive integers. For instance, if there are N object query vectors, N first object feature vectors are obtained, along with N first object mask images. From these N first object mask images, U first object mask images are determined to identify the first target objects from the first target image. Therefore, the first object feature vectors corresponding to these U first object mask images are used as the first object feature vectors corresponding to the U first target objects.

[0107] In some embodiments, determining the associated target object may involve: based on the first object feature vectors corresponding to U first target objects and the second object feature vectors corresponding to V second target objects, determining the object relevance between each first target object and each second target object; based on the object relevance, determining a second target object from the V second target objects that matches any of the U first target objects, and using the determined second target object as the associated target object. The vector similarity between the object feature vector corresponding to any first object and the object feature vector corresponding to any second object can be used as the object relevance between any first object and any second object. It can be understood that an object query vector yields a first object feature vector, and a first object feature vector corresponds to a first object mask. When the first object mask indicates that a first target object has been identified from the first target image, the first object feature vector corresponding to the first object mask is determined to be the first object feature vector corresponding to a first target object.

[0108] The process of determining a second target object matching any one of the U first target objects from V second target objects based on object relevance can be achieved by identifying a matching second target object for each of the V second target objects, and then using the Hungarian algorithm to determine the matching second target object for any one of the first target objects based on the matching second target objects for each first target object. Here, a matching second target object refers to the second target object among the V second target objects with the highest object relevance to any one of the first target objects. It can be understood that if a first target object has a matching second target object, it means that the first target object and the matching second target object are the same target object; that is, the first target object moved from its position in the first target image to its position in the second target image. In other words, the movement trajectory of each target object in the video can be obtained based on the position information of the same target objects segmented from any two adjacent video frames.

[0109] In some embodiments, generating the movement trajectory of the associated target object based on the first object mask and the second object mask may involve determining the first position information of the first target object matching the associated target object in the first target image based on the first object mask, and determining the second position information of the associated target object in the second target image based on the second object mask; and generating the movement trajectory of the associated target object using the first position information and the second position information. The first object mask corresponding to the first object feature vector includes U first object masks corresponding to the first target objects; the second object mask corresponding to the second object feature vector includes U second object masks corresponding to the second target objects.

[0110] In other words, the position information indicated by the first object mask map corresponding to the first target object that matches the associated target object in the first target image is determined as the position information of the associated target object in the first target image, and the position information indicated by the second object mask map corresponding to the associated target object in the second target image is determined as the position information of the associated target object in the second target image. The movement trajectory of the associated target object is generated using the position information of the associated target object in the first target image and its position information in the second target image. It can be understood that the object region of the first target object matching the first target object in the first target image can be determined by the first object mask map corresponding to the associated target object, thus obtaining the position information in the first target image. Similarly, the object region of the second target object matching the associated target object in the second target image can be determined by the second object mask map corresponding to the associated target object, thus obtaining the position information in the second target image. Therefore, the movement trajectory of the associated target object in the first and second target images can be generated. Simultaneously, when generating the movement trajectory, the object category to which the associated target object belongs can also be determined using the above method, and the associated target object can be displayed in the first and second target images accordingly.

[0111] For example, such as Figure 9 , Figure 9This is a schematic diagram of a target object segmentation scenario provided in an embodiment of this application; wherein, a first target image and a second target image are acquired; a first image feature vector of the first target image and a second image feature vector of the second target image are acquired; N object query features are acquired; N first object feature vectors associated with the first target image are generated based on the N object query vectors and the first image feature vectors; N second object feature vectors associated with the second target image are generated based on the N object query vectors and the second image feature vectors; N first object mask images corresponding to the first object feature vectors are generated based on the N first object feature vectors and the first image feature vectors; and N second object mask images corresponding to the second object feature vectors are generated based on the N second object feature vectors and the second image feature vectors; and the identified object is obtained from the N first object mask images. U first object masks are used to identify the first target object. The corresponding first object feature vectors for each of the U first target objects (1, 2, ..., U) are used. V second object masks are obtained from N second object masks, and the corresponding second object feature vectors for each of the V second target objects (1, 2, ..., V) are used. In other words, U first target objects are segmented from the first target image using N object query vectors, and V second target objects are segmented from the second target image. An object relevance matrix between the U first target objects and the V second target objects can be constructed based on the similarity between the first object feature vectors corresponding to each first target object and the second object feature vectors corresponding to each second target object (where, ..., V). Figure 9 In the object relevance matrix shown, "11" represents the object relevance between the first target object 1 and the second target object 2, and "12" represents the object relevance between the first target object and the second target object; the rest are similar. Using the Hungarian algorithm, a second target object matching any first target object is determined based on this object relevance matrix. For example, first target object 1 matches second target object 2, and first target object 2 matches second target object 3. Therefore, the position information of first target object 1 in the first target image can be determined based on the first object mask corresponding to first target object 1, and the position information of second target object 2 in the second target image can be determined based on the second object mask corresponding to second target object 2, to generate the movement trajectory of second target object 2. Similarly, the position information of first target object 2 in the first target image can be determined based on the first object mask corresponding to first target object 2, and the position information of second target object 3 in the second target image can be determined based on the second object mask corresponding to second target object 3, to generate the movement trajectory of second target object 3.

[0112] In this embodiment of the application, the identification logic of the object position and the object category of an object can be decoupled. The target object in the target image can be queried from the target image by using the object query vector, and the corresponding object feature vector can be generated. The object position and the same target object can be determined by the object feature vector, thereby generating the movement trajectory of the target object in the first target image and the second target image.

[0113] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in this application. It should be noted that... Figure 10 The image processing apparatus shown is used to execute this application. Figure 2 and Figure 6 The methods in the illustrated embodiments are shown only in the parts relevant to the embodiments of this application for ease of explanation; specific technical details are not disclosed. Reference to this application is required. Figure 2 and Figure 6 The illustrated embodiment. The image processing apparatus 1000 may include: a first acquisition module 1001 and a first processing module 1002. Wherein:

[0114] The first acquisition module 1001 is used to acquire a target image containing a target object and to acquire the image feature vector of the target image;

[0115] The first processing module 1002 is used to obtain an object query vector for querying target objects from a target image, and to generate an object feature vector associated with the target image based on the object query vector and the image feature vector.

[0116] The first processing module 1002 is also used to generate an object mask image corresponding to the object feature vector based on the object feature vector and the image feature vector; the object mask image is used to mark the target object;

[0117] The first processing module 1002 is further configured to obtain the category encoding vector corresponding to each object category in the object category set based on the object mask image, and determine the similarity between the object feature vector and the category encoding vector corresponding to each object category, so as to determine the object category to which the target object belongs from the object category set.

[0118] In some embodiments, the image feature vector includes R image feature sub-vectors, and one image feature sub-vector is used to characterize the image features of an image region in the target image;

[0119] The first processing module 1002, when generating object feature vectors associated with the target image based on object query vectors and image feature vectors, specifically performs the following:

[0120] The probability that the image region indicated by each image feature sub-vector belongs to the object region is determined based on the object query vector and each image feature sub-vector; the object region is used to characterize the region of the target object in the target image;

[0121] Based on the region probability indicated by each image feature vector, the R image feature vectors and the object query vector are fused to obtain the object feature vector.

[0122] In some embodiments, when the first processing module 1002 performs fusion processing on R image feature vectors and an object query vector based on the region probability indicated by each image feature vector to obtain an object feature vector, it specifically performs the following:

[0123] Based on the region probability indicated by each image feature vector, determine the feature vector to be fused corresponding to each image feature vector from each image feature vector;

[0124] The feature vector to be fused and the object query vector corresponding to each image feature sub-vector are fused to obtain the object feature vector.

[0125] In some embodiments, the image feature vector includes R image feature sub-vectors, and one image feature sub-vector is used to characterize the image features of an image region in the target image;

[0126] The first processing module 1002, when generating an object mask image corresponding to an object feature vector based on the object feature vector and the image feature vector, specifically performs the following:

[0127] Each image feature vector is convolved with the object feature vector to obtain the convolution result corresponding to each image feature vector.

[0128] The mask value corresponding to the image region indicated by each image feature vector is determined based on the convolution result corresponding to each image feature vector;

[0129] An object mask map is generated by using the mask value corresponding to the image region indicated by each image feature subvector.

[0130] In some embodiments, when the first processing module 1002 is used to obtain the category encoding vector corresponding to each object category in the object category set based on the object mask image, it is specifically used for:

[0131] If the object mask image includes target mask values ​​used to mark the target object, then obtain the category encoding vector corresponding to each object category in the object category set;

[0132] If the object mask image does not include the target mask value used to mark the target object, then the category encoding vector corresponding to each object category in the object category set is not obtained.

[0133] In some embodiments, the first processing module 1002 is further configured to:

[0134] Based on the target mask value in the object mask image, the target image is masked, and the mask region where the target mask value is located in the masked target image is taken as the object region of the target object in the target image.

[0135] The target object in the target image is highlighted according to the preset display method, and the highlighted target object and its category are displayed together.

[0136] The specific implementation methods of the first acquisition module and the first processing module can be found in the description of the above embodiments, and will not be repeated here. It should be understood that the beneficial effects obtained by using the same method will also not be repeated here.

[0137] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in this application. It should be noted that... Figure 11 The image processing apparatus shown is used to execute this application. Figure 8 The methods in the illustrated embodiments are shown only in the parts relevant to the embodiments of this application for ease of explanation; specific technical details are not disclosed. Reference to this application is required. Figure 8 The illustrated embodiment. The image processing apparatus 1100 may include: a second acquisition module 1101 and a second processing module 1102. Wherein:

[0138] The second acquisition module 1101 is used to acquire a first target image containing a first target object and a second target image containing a second target object, and to acquire the image feature vector of the first target image and the image feature vector of the second target image;

[0139] The second processing module 1102 is used to obtain an object query vector; the object query vector is used to query a first target object from a first target image and to query a second target object from a second target image.

[0140] The second processing module 1102 is further configured to generate a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and to generate a second object feature vector associated with the second target image based on the object query vector and the second image feature vector.

[0141] The second processing module 1102 is further configured to generate a first object mask image corresponding to the first object feature vector based on the first object feature vector and the first image feature vector, and to generate a second object mask image corresponding to the second object feature vector based on the second object feature vector and the second image feature vector; the first object mask image is used to mark the first target object, and the second object mask image is used to mark the second target object;

[0142] The second processing module 1102 is further configured to determine the associated target object that matches the first target object from the second target object based on the first object feature vector and the second object feature vector, and to generate the movement trajectory of the associated target object based on the first object mask map and the second object mask map.

[0143] Wherein, the first object feature vector associated with the first target image includes the first object feature vectors corresponding to U first target objects, and the second object feature vector associated with the second target image includes the second object feature vectors corresponding to V second target objects; U and V are both positive integers.

[0144] When the second processing module 1102 determines the associated target object matching the first target object from the second target object based on the first object feature vector and the second object feature vector, it specifically performs the following:

[0145] Based on the first object feature vectors corresponding to U first target objects and the second object feature vectors corresponding to V second target objects, the object correlation degree between each first target object and each second target object is determined.

[0146] Based on object relevance, a second target object is determined from V second target objects that matches any one of U first target objects, and the determined second target object is used as the associated target object.

[0147] Specifically, when the second processing module 1102 generates the movement trajectory of the associated target object based on the first object mask image and the second object mask image, it is used for:

[0148] Based on the first object mask image, determine the first position information of the first target object that matches the associated target object in the first target image, and based on the second object mask image, determine the second position information of the associated target object in the second target image;

[0149] The movement trajectory of the associated target object is generated using the first location information and the second location information.

[0150] The specific implementation methods of the second acquisition module and the second processing module can be found in the description of the above embodiments, and will not be repeated here. It should be understood that the beneficial effects obtained by using the same method will also not be repeated here.

[0151] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 12 As shown, the electronic device 1200 includes at least one processor 1201 and a memory 1202. Optionally, the electronic device may also include a network interface. The processor 1201, memory 1202, and network interface can exchange data. The network interface, controlled by the processor 1201, is used to send and receive messages. The memory 1202 stores computer programs, including program instructions. The processor 1201 executes the program instructions stored in the memory 1202. The processor 1201 is configured to invoke the program instructions to execute the aforementioned method.

[0152] The memory 1202 may include volatile memory, such as random-access memory (RAM); the memory 1202 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 1202 may also include a combination of the above types of memory.

[0153] Processor 1201 may be a central processing unit (CPU). In one embodiment, processor 1201 may also be a graphics processing unit (GPU). Processor 1201 may also be a combination of a CPU and a GPU. Processor 1201 can be used to invoke device control applications stored in memory 1202 to perform the above-described tasks. Figure 2 , Figure 6 and Figure 8 The description of the image processing method in the corresponding embodiments can also be executed as described above. Figure 10 and Figure 11 The description of the image processing apparatus in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0154] In specific implementations, the devices, processors, memory, etc., described in the embodiments of this application can execute the implementation methods described in the above method embodiments, or they can execute the implementation methods described in the embodiments of this application, which will not be repeated here.

[0155] This application also provides a computer-readable storage medium storing a computer program. The computer program includes program instructions, which, when executed by a processor, enable the processor to perform some or all of the steps described in the above method embodiments. Optionally, the computer storage medium can be volatile or non-volatile. The computer-readable storage medium may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application program required for a given function, etc.; the data storage area may store data created based on the use of blockchain nodes, etc.

[0156] This application provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it can implement some or all of the steps in the above method, which will not be elaborated here.

[0157] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0158] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer storage medium, which can be a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0159] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent changes made in accordance with the claims of this application, still fall within the scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: A first target image containing a first target object and a second target image containing a second target object are obtained, and a first image feature vector of the first target image and a second image feature vector of the second target image are obtained; the first image feature vector includes at least one first image feature sub-vector, and the first image feature sub-vector is used to characterize the image features of an image region in the first target image; the second image feature vector includes at least one second image feature sub-vector, and the second image feature sub-vector is used to characterize the image features of an image region in the second target image. Obtain an object query vector; the object query vector is used to query the first target object from the first target image and to query the second target object from the second target image; A first object feature vector associated with the first target image is generated based on the object query vector and the first image feature vector, and a second object feature vector associated with the second target image is generated based on the object query vector and the second image feature vector. The object feature vector associated with any target image is used to characterize object query information determined from any target image, and the object query information is used to indicate how to query the target object in the target image using the object query vector. Each first image feature vector is convolved with the first object feature vector to obtain the convolution result corresponding to each first image feature vector. The first object feature vector is used to determine the object associated with each first image feature vector. The first object mask is generated by using the mask values ​​corresponding to the image regions indicated by each first image feature vector; the first object mask is used to mark the first target object; each second image feature vector is convolved with a second object feature vector to obtain a convolution result corresponding to each second image feature vector; the mask value corresponding to the image region indicated by each second image feature vector is determined based on the convolution result; a second object mask is generated by using the mask values ​​corresponding to the image regions indicated by each second image feature vector; the second object mask is used to mark the second target object. Based on the first object mask map and the second object mask map, an associated target object that matches the first target object is determined from the second target object.

2. The method according to claim 1, characterized in that, The first object feature vector associated with the first target image includes U first object feature vectors corresponding to first target objects, and one first object feature vector corresponds to one first object mask image; the second object feature vector associated with the second target image includes V second object feature vectors corresponding to second target objects; one second object feature vector corresponds to one second object mask image; U and V are both positive integers; The step of determining the associated target object matching the first target object from the second target object based on the first object mask map and the second object mask map includes: Based on the first object mask map corresponding to the U first target objects and the second object mask map corresponding to the V second target objects, the object correlation degree between each first target object and each second target object is determined; Based on the object relevance, a second target object is determined from the V second target objects that matches any one of the U first target objects, and the determined second target object is used as the associated target object.

3. The method according to claim 1, characterized in that, The method further includes: The movement trajectory of the associated target object is generated based on the first object mask image and the second object mask image.

4. The method according to claim 3, characterized in that, The step of generating the movement trajectory of the associated target object based on the first object mask image and the second object mask image includes: Based on the first object mask image, determine the first position information of the first target object that matches the associated target object in the first target image, and based on the second object mask image, determine the second position information of the associated target object in the second target image; The movement trajectory of the associated target object is generated using the first location information and the second location information.

5. An image processing apparatus, characterized in that, The device includes: The second acquisition module is used to acquire a first target image containing a first target object and a second target image containing a second target object, and to acquire a first image feature vector of the first target image and a second image feature vector of the second target image; the first image feature vector includes at least one first image feature sub-vector, and the first image feature sub-vector is used to characterize the image features of an image region in the first target image; the second image feature vector includes at least one second image feature sub-vector, and the second image feature sub-vector is used to characterize the image features of an image region in the second target image. The second processing module is used to obtain an object query vector; the object query vector is used to query the first target object from the first target image and to query the second target object from the second target image. The second processing module is further configured to generate a first object feature vector associated with the first target image based on the object query vector and the first image feature vector, and to generate a second object feature vector associated with the second target image based on the object query vector and the second image feature vector; the object feature vector associated with any target image is used to characterize the object query information determined from the any target image, and the object query information is used to indicate that the target object in the target image is retrieved through the object query vector; each first image feature sub-vector is convolved with the first object feature vector to obtain the convolution result corresponding to each first image feature sub-vector; and a determination is made based on the convolution result corresponding to each first image feature sub-vector. The mask value corresponding to the image region indicated by each first image feature sub-vector; a first object mask image is generated using the mask value corresponding to the image region indicated by each first image feature sub-vector; the first object mask image is used to mark the first target object; each second image feature sub-vector is convolved using the second object feature vector to obtain the convolution result corresponding to each second image feature sub-vector; the mask value corresponding to the image region indicated by each second image feature sub-vector is determined based on the convolution result corresponding to each second image feature sub-vector; a second object mask image is generated using the mask value corresponding to the image region indicated by each second image feature sub-vector; the second object mask image is used to mark the second target object; The second processing module is further configured to determine, based on the first object mask image and the second object mask image, an associated target object that matches the first target object from the second target object.

6. An electronic device, characterized in that, The device includes a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-4.

Images