Image annotation, method, device and equipment for obtaining image annotation model

Abstract

The present application discloses a method, device and equipment for image annotation and obtaining an image annotation model, and belongs to the field of image processing technology. The method includes: obtaining a first image, in which there is at least one target object to be annotated; obtaining feature maps of multiple sizes of the first image, and the feature maps of each size in the feature maps of multiple sizes include multiple first feature vectors; selecting a first feature vector that meets a screening condition from the first feature vectors of the feature maps of multiple sizes to obtain a first feature vector corresponding to at least one target object; obtaining correlation data between the first feature vectors corresponding to at least one target object, and annotating each target object in at least one target object based on the correlation data. By selecting the first feature vector that meets the screening condition, the amount of calculation when annotating the target object is controlled, and the annotation efficiency is improved. Annotating the target object based on the correlation data between the first feature vectors improves the annotation accuracy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, and apparatus for image annotation and obtaining an image annotation model. Background Art

[0002] With the development of image processing technology, more and more application scenarios require the annotation of target objects in images. For example, in the road monitoring scenario, vehicles (target objects) in the image are annotated to facilitate the monitoring of vehicle driving regulations on the road.

[0003] In related art, when annotating an image, a rectangular area is extracted from the image, and then the positions of target objects with non-uniform shapes within the rectangular area are predicted to annotate the target objects from the image.

[0004] Since the above image annotation method can only extract rectangular areas from the image, for target objects with non-uniform shapes, the rectangular areas will introduce a lot of unnecessary background, resulting in high computational complexity and low efficiency of image annotation. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, and device for image annotation and obtaining an image annotation model, which can be used to solve problems in related technologies. The technical solution is as follows:

[0006] In one aspect, an embodiment of the present application provides a method for annotating an image, the method comprising:

[0007] Acquire a first image, wherein the first image contains at least one target object to be labeled;

[0008] Acquire feature maps of multiple sizes of the first image, wherein each feature map of the multiple sizes includes multiple first feature vectors;

[0009] Selecting a first feature vector that meets a screening condition from the first feature vectors of the feature maps of the multiple sizes, and obtaining a first feature vector corresponding to at least one target object;

[0010] Correlation data between first feature vectors corresponding to the at least one target object is obtained, and each target object in the at least one target object is labeled based on the correlation data.

[0011] In a possible implementation, selecting a first feature vector that satisfies a screening condition from the first feature vectors of the feature maps of the multiple sizes includes:

[0012] Obtaining association scores of the feature graphs of each size, where the association scores are used to reflect the association between the plurality of first feature vectors in the feature graph and the at least one target object;

[0013] A first feature vector that meets the screening condition is selected from the first feature vectors of the feature maps of the multiple sizes based on the association scores of the feature maps of the respective sizes.

[0014] In a possible implementation, selecting a first feature vector that meets the screening condition from the first feature vectors of the feature maps of the multiple sizes based on the association scores of the feature maps of the respective sizes includes:

[0015] A first number is determined based on the number of first eigenvectors of the feature graphs of the multiple sizes, the first eigenvectors of the feature graphs of the multiple sizes are arranged in order of association, and the first eigenvector arranged before the first number is selected as the first eigenvector that meets the screening condition.

[0016] In a possible implementation, the labeling each target object in the at least one target object based on the correlation data includes:

[0017] Predicting the probability that the target object corresponds to each category based on the correlation data;

[0018] The category of the target object is labeled based on the probability that the target object corresponds to each category.

[0019] In a possible implementation, the labeling each target object in the at least one target object based on the correlation data includes:

[0020] The correlation data is mapped to obtain a four-dimensional vector of the target object, and the position of the target object is marked based on the four-dimensional vector.

[0021] In a possible implementation, the method further includes:

[0022] Obtaining an image annotation model corresponding to the first image, the image annotation model comprising a feature extraction module, a screening module, and a self-attention module;

[0023] The acquiring feature maps of multiple sizes of the first image includes:

[0024] Calling the feature extraction module to obtain feature maps of multiple sizes of the first image;

[0025] The selecting a first feature vector that satisfies a screening condition from the first feature vectors of the feature maps of the multiple sizes includes:

[0026] calling the screening module to select a first feature vector that meets the screening condition from the first feature vectors of the feature maps of the multiple sizes;

[0027] The acquiring correlation data between the first feature vectors corresponding to the at least one target object, and labeling each target object in the at least one target object based on the correlation data, includes:

[0028] The self-attention module is called to obtain correlation data between the first feature vectors corresponding to the at least one target object, and each target object in the at least one target object is labeled based on the correlation data.

[0029] In a possible implementation, the feature extraction module includes multiple layers of feature extraction submodules, and the feature maps output by feature extraction submodules in different layers have different sizes. The feature map output by the feature extraction submodule in the previous layer is the input of the feature extraction submodule in the next layer.

[0030] On the other hand, an embodiment of the present application provides a method for obtaining an image annotation model, the method comprising:

[0031] Acquire a second image for training, wherein at least one training object exists in the second image;

[0032] Obtaining an initial image annotation model, wherein the initial image annotation model includes an initial feature extraction module, an initial screening module, and an initial self-attention module;

[0033] Calling the initial feature extraction module to obtain sample feature maps of multiple sizes of the second image, where each size of the sample feature maps of the multiple sizes includes multiple second feature vectors;

[0034] Calling the initial screening module to select a second feature vector that meets the initial screening condition from the second feature vectors of the sample feature maps of the multiple sizes, to obtain a second feature vector corresponding to at least one training object;

[0035] calling the initial self-attention module to obtain correlation sample data between the second eigenvectors corresponding to the at least one training object, and labeling each training object in the at least one training object based on the correlation sample data to obtain a labeling result;

[0036] Obtaining labeling information of each training object in the at least one training object, and updating the initial image labeling model based on the labeling result and the labeling information to obtain an image labeling model.

[0037] In another aspect, a device for image annotation is provided, the device comprising:

[0038] An acquisition module, configured to acquire a first image, wherein the first image contains at least one target object to be labeled;

[0039] The acquisition module is further configured to acquire feature maps of multiple sizes of the first image, wherein each feature map of the multiple sizes includes multiple first feature vectors;

[0040] A selection module, configured to select a first feature vector that meets a screening condition from the first feature vectors of the feature maps of the multiple sizes, and obtain a first feature vector corresponding to at least one target object;

[0041] The labeling module is configured to obtain correlation data between the first feature vectors corresponding to the at least one target object, and label each target object in the at least one target object based on the correlation data.

[0042] In one possible implementation, the selection module is used to obtain the association scores of the feature graphs of each size, where the association scores are used to reflect the association between multiple first feature vectors in the feature graph and the at least one target object; and select the first feature vector that meets the screening condition from the first feature vectors of the feature graphs of the multiple sizes based on the association scores of the feature graphs of each size.

[0043] In one possible implementation, the selection module is used to determine a first quantity based on the number of first eigenvectors of the feature graphs of the multiple sizes, arrange the first eigenvectors of the feature graphs of the multiple sizes in order of association, and select the first eigenvector arranged before the first quantity as the first eigenvector that meets the screening condition.

[0044] In a possible implementation, the labeling module is configured to predict the probability that the target object corresponds to each category based on the correlation data; and label the category of the target object based on the probability that the target object corresponds to each category.

[0045] In a possible implementation, the labeling module is configured to map the correlation data to obtain a four-dimensional vector of the target object, and label the position of the target object based on the four-dimensional vector.

[0046] In one possible implementation, the acquisition module is also used to obtain an image annotation model corresponding to the first image, and the image annotation model includes a feature extraction module, a screening module and a self-attention module; the feature extraction module is called to obtain feature maps of multiple sizes of the first image; the selection module is also used to call the screening module to select the first feature vector that meets the screening condition from the first feature vectors of the feature maps of the multiple sizes; the annotation module is also used to call the self-attention module to obtain correlation data between the first feature vectors corresponding to the at least one target object, and annotate each target object in the at least one target object based on the correlation data.

[0047] In a possible implementation, the feature extraction module includes multiple layers of feature extraction submodules, and the feature maps output by feature extraction submodules in different layers have different sizes. The feature map output by the feature extraction submodule in the previous layer is the input of the feature extraction submodule in the next layer.

[0048] In another aspect, a device for obtaining an image annotation model is provided, the device comprising:

[0049] an acquisition module, configured to acquire a second image for training, wherein the second image contains at least one training object;

[0050] The acquisition module is further used to acquire an initial image annotation model, which includes an initial feature extraction module, an initial screening module and an initial self-attention module;

[0051] The acquisition module is further configured to call the initial feature extraction module to acquire sample feature maps of multiple sizes of the second image, wherein each size of the sample feature maps of the multiple sizes includes multiple second feature vectors;

[0052] A selection module, configured to call the initial screening module to select a second feature vector that meets an initial screening condition from the second feature vectors of the sample feature maps of the multiple sizes, and obtain a second feature vector corresponding to at least one training object;

[0053] a labeling module, configured to call the initial self-attention module to obtain correlation sample data between the second eigenvectors corresponding to the at least one training object, and label each training object in the at least one training object based on the correlation sample data to obtain a labeling result;

[0054] An updating module is configured to obtain labeling information of each training object in the at least one training object, and update the initial image labeling model based on the labeling result and the labeling information to obtain an image labeling model.

[0055] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer device implements any of the above-mentioned image annotation methods or any of the above-mentioned methods for obtaining an image annotation model.

[0056] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above-mentioned image annotation methods or any of the above-mentioned methods for obtaining an image annotation model.

[0057] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the aforementioned methods for image annotation or implement any of the aforementioned methods for obtaining an image annotation model.

[0058] The technical solutions provided by the embodiments of the present application bring at least the following beneficial effects:

[0059] When annotating an image, by selecting the first eigenvector that meets the screening criteria as the first eigenvector corresponding to the target object, the computational effort required for subsequent object annotation is effectively reduced, improving annotation efficiency. Furthermore, when annotating the target object based on the first eigenvector, the correlation data between each first eigenvector is taken into account, improving annotation accuracy.

[0060] In the process of obtaining the image annotation model, by screening the second eigenvector, the amount of computation when the initial self-attention module annotates the training object is effectively controlled, thereby improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0063] Figure 2 This is a flow chart of an image annotation method provided by an embodiment of the present application;

[0064] Figure 3 This is a schematic diagram of a process for image annotation using an image annotation model provided in an embodiment of the present application;

[0065] Figure 4This is a flow chart of a method for obtaining an image annotation model provided in an embodiment of the present application;

[0066] Figure 5 This is a schematic diagram of an interface of an image annotation result provided in an embodiment of the present application;

[0067] Figure 6 Schematic diagram of the structure of an image annotation device provided in an embodiment of the present application;

[0068] Figure 7 Schematic diagram of a device for obtaining an image annotation model provided in an embodiment of the present application;

[0069] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of the present application;

[0070] Figure 9 This is a structural diagram of a network device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0072] This application embodiment provides a method for image annotation and obtaining an image annotation model. Figure 1 , which shows a schematic diagram of an implementation environment of the method provided in an embodiment of the present application. The implementation environment may include: a terminal 11 and a server 12.

[0073] Among them, the terminal 11 and the server 12 can independently implement the method for image annotation and obtaining the image annotation model provided in the embodiment of the present application. The terminal 11 and the server 12 can also implement the method for image annotation and obtaining the image annotation model provided in the embodiment of the present application through interaction. For example, the terminal 11 obtains a first image, sends the obtained first image to the server 12, and the server 12 can apply the method provided in the embodiment of the present application to annotate the target object in the first image. Alternatively, the terminal 11 obtains a second image, sends the obtained second image to the server 12, and the server 12 can apply the method provided in the embodiment of the present application to obtain the image annotation model, and send the image annotation model to the terminal 11. The terminal 11 obtains the first image and annotates the target object in the first image based on the image annotation model.

[0074] Optionally, the terminal 11 can be any electronic product that can interact with the user through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a tablet computer, a smart car computer, a smart TV, a smart speaker, etc. The server 12 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. The terminal 11 establishes a communication connection with the server 12 via a wired or wireless network.

[0075] Those skilled in the art should understand that the above-mentioned terminal 11 and server 12 are only examples. Other existing or future terminals or servers that are applicable to this application should also be included in the scope of protection of this application and are included here by reference.

[0076] Based on the above Figure 1 In the implementation environment shown, the present application embodiment provides a method for image annotation. Figure 2 As shown, the image annotation method can be executed by a terminal or a server. Taking the method applied to a terminal as an example, the method includes steps 201 to 204.

[0077] In step 201 , a first image is acquired, in which there is at least one target object to be labeled.

[0078] The embodiment of the present application does not limit the manner in which the terminal obtains the first image, and the first image may be obtained in the following three ways, including but not limited to.

[0079] Acquisition method 1: Acquire a first image based on an image acquisition device.

[0080] Optionally, the image acquisition device can be set anywhere, can be set on the terminal, or can be set outside the terminal. For example, the image acquisition device is a camera set on both sides of the road for photographing passing vehicles. Exemplarily, when the image acquisition device is set outside the terminal, the terminal communicates with the image acquisition device, acquires images based on the communication connection, and obtains a first image. Among them, the communication connection between the terminal and the image acquisition device can be directly implemented through a wired or wireless network, or can be implemented based on other network devices. Taking the image acquisition device as a camera set on both sides of the road as an example, after the camera takes the image, it uploads the captured image to the server (other network device) of the road management center, and the server of the road management center stores the image in the storage space. The terminal accesses the storage space based on the communication connection with the server of the road management center, and uses the image obtained by accessing the storage space as the first image. When the image acquisition device is set on the terminal, the terminal can take an image based on the image acquisition device and use the captured image as the first image.

[0081] Furthermore, the image captured by the image capture device may be a static image. In this case, the terminal may use the captured image as the first image, or may crop the static image and use the cropped static image as the first image. Of course, the image captured by the image capture device may also be a non-static image, such as a video. In this case, the terminal may capture an image from the non-static image and use the captured image as the first image.

[0082] Acquisition method 2: Acquire the first image from the image database.

[0083] Optionally, the image database is used to provide images to meet business needs. Business needs may include requirements for training algorithms, statistical analysis, or other requirements. The terminal accesses the image database and obtains an image stored in the image database as the first image. The images in the image database may be acquired using an image acquisition device or other methods, and this is not limited in the present embodiment.

[0084] Acquisition method three: the terminal accesses the storage space to acquire the first image.

[0085] Optionally, at least one image is stored in the storage space of the terminal, and the terminal selects an image from the at least one image stored in the storage space as the first image. The images in the storage space may be acquired and stored in the storage space using the two acquisition methods described above, or acquired and stored in the storage space using other methods, and this embodiment of the present application is not limited thereto.

[0086] It should be noted that no matter how the terminal obtains the first image, there is at least one target object to be annotated in the first image. In one possible implementation, the purpose of image annotation is to annotate a small target in the first image. At this time, the target object refers to an object with a resolution less than the target threshold, that is, the small target is the target object. Exemplarily, the target threshold is an arbitrary value set based on experience, for example, the target threshold is set to 30×30. The first image with a resolution less than 30×30 belongs to the target object to be annotated. The embodiment of the present application does not limit the category of the target object in the first image. The category of the target object can be a vehicle, a pedestrian, a billboard, or other types. The number of categories of target objects in an image can be one or more. For example, it is determined that the vehicles and pedestrians in the first image are target objects to be annotated.

[0087] In step 202 , feature maps of multiple sizes of a first image are obtained, where each feature map of the multiple sizes includes multiple first feature vectors.

[0088] In one possible implementation, before acquiring feature maps of multiple sizes of the first image, a preprocessing operation is performed on the first image to improve the quality of the first image and thereby improve the accuracy of the annotation results of the first image. For example, the preprocessing operation may include performing angle correction on the first image or performing denoising on the first image.

[0089] The embodiments of the present application do not limit the process of obtaining feature maps of multiple sizes. Optionally, the first image is scaled to multiple scales to obtain images of different sizes, and the first feature vector is extracted from the images of different sizes to obtain feature maps of multiple sizes including the first feature vector. Since the feature information included in feature maps of different sizes is different, for example, the spatial information of a feature map of a small size is higher than that of a feature map of a large size, and the semantic information of a feature map of a large size is higher than that of a feature map of a small size, by obtaining feature maps of different sizes, richer feature information is obtained.

[0090] Exemplarily, when the terminal obtains an image annotation model corresponding to the first image before annotating the first image, and the image annotation model includes a feature extraction module, a screening module, and a self-attention module, the terminal calls the feature extraction module to obtain feature maps of multiple sizes of the first image. The image annotation model corresponding to the first image refers to an image annotation model used to extract target objects present in the first image. For example, if the target objects present in the first image are vehicles and pedestrians, the image annotation model corresponding to the first image is also the model used to annotate pedestrians and vehicles in the first image.

[0091] In one possible implementation, the feature extraction module includes multiple layers of feature extraction submodules. The feature maps output by feature extraction submodules in different layers have different sizes. The feature map output by the feature extraction submodule in the previous layer serves as the input to the feature extraction submodule in the next layer.

[0092] Optionally, the feature extraction module is FPN (Feature Pyramid Networks). In this case, the feature extraction submodule is each layer of the FPN network. Figure 3 A schematic diagram of a process for image annotation using an image annotation model provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the image annotation model includes a feature extraction module 301, a screening module 302, and a self-attention module 303. After the first image is input into the feature extraction module, the feature extraction module sequentially performs four downsampling operations on the first image. The output of the previous downsampling is used as the input of the next downsampling. The size of the first image is reduced by downsampling, and the first feature vector at the size is extracted based on the reduced first image. After performing the four downsampling operations to obtain the fourth layer feature map (304), the fourth layer feature map is sequentially performed three upsampling operations (enlarging the size of the feature map).

[0093] It should be noted that when the fourth-layer feature map is upsampled three times, it is horizontally connected to the corresponding feature map of the same size, thereby fusing the feature information of the feature maps of each layer to obtain feature maps of different sizes with stronger expressiveness. Among them, the horizontal connection can be achieved by adding pixels. Stronger expressiveness means that the feature information included in the feature map is richer. Regarding the feature map and the corresponding feature map of the same size, taking the first upsampling operation as an example (305), the feature map of the same size corresponding to the feature map at this time is the third-layer feature map (306) obtained by the downsampling operation.

[0094] See also Figure 3 When four feature maps of different sizes are obtained through the above-mentioned upsampling operation, two downsampling operations are performed to obtain feature maps of smaller sizes, so that the detection cell covers a wider area, thereby detecting the target object with higher resolution in the first image. In other words, feature maps of six sizes are obtained through the feature extraction module. Optionally, the feature extraction module is composed of a ResNet (Residual Network), and different feature extraction submodules have different numbers of residual modules, thereby obtaining feature maps of different sizes.

[0095] It should be noted that the above examples are intended to illustrate the process of obtaining feature maps of different sizes, rather than to limit the process of obtaining feature maps of different sizes. The terminal can choose to obtain feature maps of different sizes of the first image through a network model, or can choose other methods. The network model can be the feature extraction module shown in the above embodiment, or it can be another model, or it can be composed of multiple models.

[0096] In addition, for the order of upsampling and downsampling operations, the terminal can select Figure 3 As shown, 4 downsampling operations are performed first, and then 2 downsampling operations are performed while performing the upsampling operation. The terminal can also choose to directly perform 6 downsampling operations, and then perform 5 upsampling operations on the feature maps obtained by the 6 downsampling operations. Of course, the number of feature maps of different sizes obtained can be 6 as shown in the above example, or other numbers, and the embodiments of the present application are not limited to this.

[0097] In step 203 , a first feature vector that meets a screening condition is selected from the first feature vectors of feature maps of multiple sizes to obtain a first feature vector corresponding to at least one target object.

[0098] In one possible implementation, a method for selecting a first feature vector that meets a filtering condition from first feature vectors of feature graphs of multiple sizes includes, but is not limited to: obtaining association scores of feature graphs of each size, where the association scores are used to reflect the degree of association between multiple first feature vectors in the feature graph and at least one target object; and selecting a first feature vector that meets the filtering condition from the first feature vectors of feature graphs of multiple sizes based on the association scores of the feature graphs of each size.

[0099] Optionally, the process of selecting the first eigenvector that meets the screening condition includes: determining the first quantity based on the number of first eigenvectors of feature maps of multiple sizes, arranging the first eigenvectors of feature maps of multiple sizes in order of correlation, and selecting the first eigenvector arranged before the first quantity as the first eigenvector that meets the screening condition. For example, feature fusion of feature maps of different sizes to obtain a target feature map; determining the first quantity based on the number of first eigenvectors in the target feature map. Optionally, the first quantity is determined based on experience based on the number of first eigenvectors in the target feature map. Taking the example of arranging in sequence from large to small according to the correlation, the target feature map includes 10,000 first eigenvectors, and the first number of first eigenvectors corresponding to the target object is set to 2,000 based on experience. At this time, the first 2,000 first eigenvectors in the arrangement are selected. Of course, the first quantity can also be determined by setting a screening ratio based on experience, and determining it based on the number of first eigenvectors in the target feature map and the screening ratio. Taking the order of arrangement from large to small according to the correlation as an example, the screening ratio is set to 1 / 5 based on experience. The target feature map includes 10,000 first eigenvectors. At this time, the first number is 2,000, and the first 2,000 first eigenvectors are selected.

[0100] It should be noted that in addition to the above example of determining the first number and thus determining the screening condition based on the number of first eigenvectors of feature maps of multiple sizes, the screening condition can also be set based on experience. For example, a correlation threshold is set based on experience, and the first eigenvector in the target feature map with a correlation degree higher than the correlation threshold is selected as the first eigenvector corresponding to the target object.

[0101] In one possible implementation, Figure 3 The image annotation model shown can call the screening module to select the first eigenvector that meets the screening conditions from the first eigenvectors of feature maps of multiple sizes. Optionally, the screening module is composed of a multi-layer convolutional network, and the score map (score map) of the feature maps of different sizes is obtained by inputting feature maps of different sizes into their corresponding screening modules. Among them, the score map is used to indicate the probability that the point is the target object, that is, the score map is used to reflect the association score of the degree of association between the first eigenvector and the target object. By feature fusion of multiple scoremaps, the top (highest) k (first number) first eigenvectors are selected as the first eigenvectors corresponding to the target object. Of course. The screening module can also be other network structures, which are not limited in the embodiments of the present application.

[0102] In step 204 , correlation data between first feature vectors corresponding to at least one target object is obtained, and each target object in the at least one target object is labeled based on the correlation data.

[0103] In one possible implementation, by interacting the first eigenvector corresponding to at least one target object, the correlation between the first eigenvectors corresponding to at least one target object is obtained to obtain correlation data. Regarding the interaction of the first eigenvector corresponding to at least one target object, optionally, the spatial position of the first eigenvector corresponding to the target object is encoded to obtain a first matrix composed of the first eigenvectors corresponding to the target object; based on the first matrix, a second matrix including the correlation between the first eigenvectors corresponding to at least one target object is obtained, which is the correlation data. By obtaining the correlation data, when subsequently labeling the target object based on the first eigenvector, auxiliary judgment can also be made based on the correlation data between multiple first eigenvectors, thereby improving the accuracy of the labeling results.

[0104] The embodiment of the present application does not limit the method of labeling each target object in at least one target object based on correlation data, and can include but is not limited to the following two methods.

[0105] Labeling method 1: Predict the probability of the target object corresponding to each category based on correlation data; label the category of the target object based on the probability of the target object corresponding to each category.

[0106] Taking the correlation data shown in the above embodiment as the second matrix as an example, the second matrix is ​​mapped using a linear layer to obtain a score vector for classification, where the score vector can predict the probability of the target object corresponding to each category, and the category of the target object is determined based on the score vector, thereby marking the category of the target object.

[0107] The second marking method maps the correlation data to obtain a four-dimensional vector of the target object, and marks the position of the target object based on the four-dimensional vector.

[0108] Taking the correlation data shown in the above embodiment as the second matrix as an example, the second matrix is ​​mapped using a linear layer to obtain a four-dimensional vector. Optionally, the four-dimensional vector refers to the coordinates of the bounding box used to mark the target object.

[0109] It should be noted that the above examples are intended to illustrate the target object labeling process, rather than to limit the target object labeling process. Since the mapping correlation data can obtain the probability of the target object corresponding to each category and the four-dimensional vector of the target object, that is, the position and category of the target object can be labeled based on the correlation data. Therefore, the terminal can choose to label either the position or category of the target object, or can label the position and category of the target object at the same time. This embodiment of the present application does not limit this.

[0110] In one possible implementation, see Figure 3In the image annotation model shown, the process of annotating the target object includes: calling the self-attention module to obtain correlation data between the first eigenvectors corresponding to at least one target object, and annotating each target object in the at least one target object based on the correlation data. Optionally, taking the correlation data as a second matrix as an example, the annotation process based on the self-attention module is as follows: encoding the spatial positions of the first eigenvectors corresponding to the target objects to obtain a first matrix composed of the first eigenvectors corresponding to the target objects; inputting the first matrix into the self-attention module, outputting a second matrix based on the self-attention module including the correlations between the first eigenvectors corresponding to at least one target object; mapping the second matrix, and annotating the target objects based on the mapping results.

[0111] For example, when labeling the target object based on the self-attention module, a multi-head attention method is used. Multi-head attention allows the self-attention module to focus on different first eigenvectors, so that when obtaining the correlation data between the first eigenvectors, the correlation between the first eigenvectors that have been noticed before can be taken into account, and the correlation between different first eigenvectors can be fully utilized. For the acquisition process of the image annotation model corresponding to the first image, please refer to Figure 4 The embodiments shown are not described in detail here.

[0112] Optionally, after marking the target object in the first image, the terminal will also display the marked first image (307). Figure 3 As shown, the rectangular box in 307 is a bounding box determined based on a four-dimensional vector, and the position of the target object is marked by the bounding box. In addition, in the case of marking the category of the target object, the terminal can display the category on the target object. In one possible implementation, the category includes category and score. By calculating the score, finding the correspondence between the score and the category, the category is determined. Exemplarily, when the score is in the range of 0.7-0.9, the category is 4, and 4 is used to indicate a small car. For example, category: 4; score: 0.78. Among them, the category can be marked inside the bounding box or outside the bounding box. The category can be marked in the upper left corner of the bounding box, or it can be marked in other positions, and this embodiment of the present application does not limit this. Regarding the correspondence between score and category, the correspondence between category and category can be determined during the training process of the image annotation model.

[0113] In one possible implementation, when the first image does not include the target object, or the category of the target object included in the first image is different from the object category that can be annotated by the image annotation model, there is no bounding box and category annotation on the output first image, that is, the first image before annotation is the same as the first image after annotation.

[0114] It should be noted that the process of annotating images based on the image annotation model can be as follows Figure 3 As shown, a first image (308) is input to the image annotation model, and the feature extraction module, the screening module, and the self-attention module output the annotated first image (307). Alternatively, the image annotation can be completed by executing the above steps in sequence through multiple independent models, which is not limited in this embodiment of the present application.

[0115] In summary, the image annotation method provided in the embodiment of the present application effectively controls the amount of subsequent calculations when annotating the target object by selecting the first eigenvector that meets the screening conditions as the first eigenvector corresponding to the target object, thereby improving the annotation efficiency. In addition, when annotating the target object based on the first eigenvector, the correlation data between the various first eigenvectors is taken into account, and the target object is assisted in annotation based on the correlation data between the various first eigenvectors, thereby improving the accuracy of annotation. By obtaining the correlation data between different first eigenvectors, a strong interaction between the first eigenvectors corresponding to the target object is achieved, which reduces the calculation on the redundant background and thereby improves the speed of annotating the target object.

[0116] Based on the above Figure 1 In the implementation environment shown, the present application embodiment provides a method for obtaining an image annotation model, such as Figure 4 As shown, taking the method applied to a terminal as an example, the method includes steps 401 to 406.

[0117] In step 401 , a second image for training is acquired, where at least one training object exists in the second image.

[0118] For example, the process of acquiring the second image is the same as above. Figure 2 The acquisition process of the first image in the embodiment shown in step 201 is similar and will not be elaborated here. It should be noted that, since the category of the training object in the second image is related to the role of the image annotation model, the annotation requirements must also be considered when acquiring the second image, and the second image is acquired based on the annotation requirements. Among them, the annotation requirements refer to the category of the object annotated using the image annotation model. When acquiring the second image, it is ensured that the category of the training object in the second image is consistent with the category of the object that needs to be annotated using the image annotation model. For example, when the role of the image annotation model is to annotate the target object in the first image, the category of the training object in the second image is consistent with the category of the target object.

[0119] In step 402, an initial image annotation model is obtained, where the initial image annotation model includes an initial feature extraction module, an initial screening module, and an initial self-attention module.

[0120] The embodiment of the present application does not limit the network architecture of the obtained initial image annotation model. It can be composed of an initial feature extraction module consisting of a residual network, an initial screening module and an initial self-attention module consisting of a convolutional neural network, or other networks with the same functions.

[0121] In step 403, an initial feature extraction module is called to obtain sample feature maps of multiple sizes of the second image, where each size of the sample feature maps of the multiple sizes includes multiple second feature vectors.

[0122] For example, the process of calling the initial feature extraction module to obtain sample feature maps of multiple sizes is the same as the above Figure 2 The process of calling the feature extraction module to obtain feature maps of multiple sizes in the embodiment shown in step 202 is similar and will not be described in detail here.

[0123] In step 404, an initial screening module is called to select a second feature vector that meets the initial screening condition from the second feature vectors of the sample feature maps of multiple sizes, and obtain a second feature vector corresponding to at least one training object.

[0124] For example, the process of determining the initial screening conditions is the same as above. Figure 2 The process of determining the screening conditions in the embodiment shown in step 203 is similar. The process of calling the initial screening module to select the second feature vector that meets the initial screening conditions is the same as the above Figure 2 The process of calling the screening module to select the first feature vector that meets the screening condition in the embodiment shown in step 203 is similar and will not be described in detail here.

[0125] In step 405, the initial self-attention module is called to obtain correlation sample data between the second feature vectors corresponding to at least one training object, and each training object in the at least one training object is labeled based on the correlation sample data to obtain a labeling result.

[0126] For example, the process of calling the initial self-attention module to label each training object is the same as above. Figure 2 The process of calling the self-attention module to label each target object in the embodiment shown in step 204 is similar and will not be repeated here.

[0127] In step 406 , labeling information of each training object in the at least one training object is obtained, and the initial image labeling model is updated based on the labeling result and the labeling information to obtain an image labeling model.

[0128] The embodiment of the present application does not limit the method of obtaining the annotation information, and it can be to manually annotate each training object to obtain the annotation information. It can also be to access the image database to obtain the images used to train the image annotation algorithm and the annotation information of the images, that is, when obtaining the second image, the annotation information of each training object in the second image is obtained at the same time. It can also be to use a reference model with a complex network structure that has a long running time but meets the annotation accuracy requirements of the training, and obtain the annotation information of the second image through the reference model. Exemplarily, the reference model can be ResNeXt (a classification model) or Swin Transformer (a target detection model). The annotation information includes at least one of position information and category information.

[0129] Optionally, a labeling loss of the initial image labeling model when labeling the second image is calculated based on the labeling information and the labeling results. Supervised learning is performed based on the labeling loss, and parameters of each module in the initial image labeling model are updated to obtain an image labeling model. The labeling loss can be calculated based on a loss function, exemplarily a cross-entropy loss function.

[0130] It should be noted that, in addition to the above-mentioned example of updating the parameters of each module in the initial image annotation model based on the annotation loss to obtain the image annotation model. It is also possible to train each module in the initial image annotation model separately to obtain the image annotation model. Optionally, when training the initial screening module, a target mask containing the training object is obtained, and the screening loss between the score map output by the initial screening module and the mask is used to implement supervised learning, update the parameters of the initial screening module, and obtain the screening module. The process of obtaining the mask is similar to the process of obtaining the annotation information shown in the above embodiment, and the process of obtaining the screening loss is similar to the process of obtaining the annotation loss shown in the above embodiment, and will not be elaborated here.

[0131] Of course, when training each module, the terminal can choose to train it individually or sequentially. For example, sequential training includes training the initial feature extraction module, outputting sample feature maps of multiple sizes of the training image based on the trained feature extraction module, training the initial screening module based on the sample feature maps, outputting a filtered second feature vector based on the trained screening module, and training the initial self-attention module based on the filtered second feature vector to obtain a self-attention module.

[0132] In summary, the method for obtaining an image annotation model provided in the embodiments of the present application effectively controls the computational effort required by the initial self-attention module to annotate training objects by screening the second eigenvector, thereby improving training efficiency. The trained image annotation model employs a more streamlined network process to implement image annotation, resulting in fast and highly accurate annotation based on the trained image annotation model.

[0133] In one possible implementation, Figure 4 The image annotation model obtained in the illustrated embodiment can be applied to the task of annotating vehicles and pedestrians. Figure 5 , where 501 is the first image before annotation, and 502 is the first image after annotation.

[0134] The above method can effectively improve the performance in the task of annotating vehicles and pedestrians. The optimization results are shown in Table 1:

[0135] Table 1

[0136]

[0137] As can be seen from Table 1, the present application solution is compared with QueryDet from the perspectives of AP, AP50, AP75, AR1, AR10, AR100, AR500 and FPS. Among them, the division of positive and negative samples is achieved based on IoU (Intersection over Union), and samples with IoU greater than the IoU threshold are regarded as positive samples, that is, samples with correct predictions. IoU is used to indicate the degree of overlap between the output box predicted by the model and the true marked box (annotation information). The data in the AP column in Table 1 above is the average of the precision of each class in multi-class predictions. The 50 in AP50 means that the IoU threshold is 0.5, and the 75 in AP75 means that the IoU threshold is 0.75. The data in the AR column is the proportion of targets predicted correctly by the model among all target objects. The "1" in AR1 indicates that a maximum of 1 output box can be output for each image annotation. The "10" in AR10 indicates that a maximum of 10 output boxes can be output for each image annotation. The "100" in AR100 indicates that a maximum of 100 output boxes can be output for each image annotation. The "500" in AR500 indicates that a maximum of 500 output boxes can be output for each image annotation. The "FPS" column indicates the number of images that can be annotated per second, which is used to evaluate the image annotation speed.

[0138] Based on the values ​​in Table 1, it can be seen that the image annotation model trained by the technical solution provided in the embodiment of the present application has better performance in the task of annotating vehicles and pedestrians.

[0139] See also Figure 6, an embodiment of the present application provides an image annotation device, which includes: an acquisition module 601, a selection module 602 and a annotation module 603.

[0140] An acquisition module 601 is configured to acquire a first image, wherein the first image contains at least one target object to be labeled;

[0141] The acquisition module 601 is further configured to acquire feature maps of multiple sizes of the first image, wherein each feature map of the multiple sizes includes multiple first feature vectors.

[0142] A selection module 602 is configured to select a first feature vector that meets a screening condition from the first feature vectors of feature maps of multiple sizes, and obtain a first feature vector corresponding to at least one target object;

[0143] The labeling module 603 is configured to obtain correlation data between first feature vectors corresponding to at least one target object, and label each target object in the at least one target object based on the correlation data.

[0144] Optionally, the selection module 602 is used to obtain the association scores of the feature maps of each size, and the association scores are used to reflect the association between multiple first feature vectors in the feature map and at least one target object; based on the association scores of the feature maps of each size, the first feature vector that meets the screening conditions is selected from the first feature vectors of the feature maps of multiple sizes.

[0145] Optionally, the selection module 602 is used to determine the first quantity based on the number of first eigenvectors of feature graphs of multiple sizes, arrange the first eigenvectors of feature graphs of multiple sizes in order of association, and select the first eigenvector arranged before the first quantity as the first eigenvector that meets the screening condition.

[0146] Optionally, the labeling module 603 is configured to predict the probability that the target object corresponds to each category based on the correlation data; and label the category of the target object based on the probability that the target object corresponds to each category.

[0147] Optionally, the labeling module 603 is configured to map the correlation data to obtain a four-dimensional vector of the target object, and label the position of the target object based on the four-dimensional vector.

[0148] Optionally, the acquisition module 601 is also used to obtain an image annotation model corresponding to the first image, and the image annotation model includes a feature extraction module, a screening module and a self-attention module; the feature extraction module is called to obtain feature maps of multiple sizes of the first image; the selection module 602 is also used to call the screening module to select the first feature vector that meets the screening conditions from the first feature vectors of the feature maps of multiple sizes; the annotation module 603 is also used to call the self-attention module to obtain correlation data between the first feature vectors corresponding to at least one target object, and annotate each target object in the at least one target object based on the correlation data.

[0149] Optionally, the feature extraction module includes multiple layers of feature extraction submodules, and the feature maps output by the feature extraction submodules in different layers have different sizes, and the feature map output by the feature extraction submodule in the previous layer is the input of the feature extraction submodule in the next layer.

[0150] When annotating an image, the above-described device selects the first eigenvector that meets the screening criteria as the first eigenvector corresponding to the target object, effectively reducing the computational effort required to annotate the target object and improving annotation efficiency. Furthermore, when annotating the target object based on the first eigenvector, the device also considers the correlation data between the various first eigenvectors, using this correlation data to assist in the annotation of the target object, thereby improving annotation accuracy.

[0151] See also Figure 7 , an embodiment of the present application provides a device for obtaining an image annotation model, which includes: an acquisition module 701, a selection module 702, a annotation module 703 and an update module 704.

[0152] An acquisition module 701 is configured to acquire a second image for training, wherein the second image contains at least one training object;

[0153] The acquisition module 701 is further used to acquire an initial image annotation model, which includes an initial feature extraction module, an initial screening module and an initial self-attention module;

[0154] The acquisition module 701 is further configured to call the initial feature extraction module to acquire sample feature maps of multiple sizes of the second image, wherein each of the sample feature maps of the multiple sizes includes multiple second feature vectors;

[0155] A selection module 702 is configured to call an initial screening module to select a second feature vector that meets an initial screening condition from the second feature vectors of sample feature maps of multiple sizes, and obtain a second feature vector corresponding to at least one training object;

[0156] A labeling module 703 is configured to call an initial self-attention module to obtain correlation sample data between second eigenvectors corresponding to at least one training object, and label each training object in the at least one training object based on the correlation sample data to obtain a labeling result;

[0157] The updating module 704 is configured to obtain the labeling information of each training object in the at least one training object, and update the initial image labeling model based on the labeling result and the labeling information to obtain an image labeling model.

[0158] In the process of acquiring the image annotation model, the above-mentioned device effectively controls the amount of calculation when the initial self-attention module annotates the training object by screening the second eigenvector, thereby improving the training efficiency.

[0159] It should be noted that the apparatus provided in the above embodiments is merely illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0160] Figure 8 801 and 802. The server may have different configurations or performances, and may include one or more processors (CPUs) 801 and one or more memories 802. The one or more memories 802 store at least one computer program, which is loaded and executed by the one or more processors 801 to enable the server to achieve the above-mentioned Figure 2 The method embodiment shown in the figure provides an image annotation method, or implements the above Figure 4 The method for obtaining an image annotation model provided by the method embodiment described above. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server may also include other components for realizing device functions, which will not be described in detail here.

[0161] Figure 9The following is a schematic diagram of a network device structure provided in an embodiment of the present application. The device may be a terminal, such as a smartphone, tablet computer, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, laptop computer, or desktop computer. A terminal may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar terminology.

[0162] Typically, the terminal includes: a processor 901 and a memory 902 .

[0163] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0164] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, which is executed by the processor 901 to enable the terminal to implement the above-mentioned Figure 2 The method embodiment shown in the figure provides an image annotation method, or implements the above Figure 4The method embodiment provides a method for obtaining an image annotation model.

[0165] In some embodiments, the terminal may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0166] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0167] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0168] Display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 905 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 905. These touch signals can be input as control signals to processor 901 for processing. Display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 905, located on the front panel of the terminal. In other embodiments, there can be at least two display screens 905, located on different surfaces of the terminal or in a foldable design. In still other embodiments, display screen 905 can be a flexible display, located on a curved or foldable surface of the terminal. Display screen 905 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0169] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0170] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, each disposed at different parts of the terminal. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0171] Positioning component 908 is used to locate the current geographic location of the terminal to implement navigation or LBS (Location Based Service). Positioning component 908 can be based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja system, or the European Union's Galileo system.

[0172] Power supply 909 is used to power various components in the terminal. Power supply 909 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 909 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0173] In some embodiments, the terminal further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

[0174] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect game or user motion data.

[0175] The gyroscope sensor 912 can detect the terminal's body orientation and rotation angle. It can also work with the accelerometer 911 to collect 3D motions of the user on the terminal. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0176] The pressure sensor 913 can be set in the side frame of the terminal and / or the lower layer of the display screen 905. When the pressure sensor 913 is set in the side frame of the terminal, it can detect the user's grip signal of the terminal, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set in the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0177] The fingerprint sensor 914 is used to collect the user's fingerprint, and the processor 901 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 901 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 can be set on the front, back, or side of the terminal. When a physical button or manufacturer logo is provided on the terminal, the fingerprint sensor 914 can be integrated with the physical button or manufacturer logo.

[0178] The optical sensor 915 is used to detect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity detected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity detected by the optical sensor 915.

[0179] Proximity sensor 916, also known as a distance sensor, is typically located on the front panel of the terminal. Proximity sensor 916 is used to detect the distance between the user and the front of the terminal. In one embodiment, when proximity sensor 916 detects that the distance between the user and the front of the terminal is gradually decreasing, processor 901 controls display screen 905 to switch from the screen-on state to the screen-off state. When proximity sensor 916 detects that the distance between the user and the front of the terminal is gradually increasing, processor 901 controls display screen 905 to switch from the screen-off state to the screen-on state.

[0180] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the network device, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0181] In an exemplary embodiment, a computer device is also provided, comprising a processor and a memory, wherein the memory stores at least one computer program. The at least one computer program is loaded and executed by one or more processors to cause the computer device to implement any of the aforementioned image annotation methods or any of the aforementioned methods for obtaining an image annotation model.

[0182] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor of a computer device to enable the computer to implement any of the above-mentioned image annotation methods, or any of the above-mentioned methods for obtaining an image annotation model.

[0183] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0184] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the aforementioned image annotation methods or any of the aforementioned methods for obtaining an image annotation model.

[0185] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first and second images involved in this application were both obtained with full authorization.

[0186] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0187] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for image annotation, characterized in that: The method comprises: Acquire a first image, wherein the first image contains at least one target object to be labeled; Acquire feature maps of multiple sizes of the first image, wherein each feature map of the multiple sizes includes multiple first feature vectors; Selecting a first feature vector that meets a screening condition from the first feature vectors of the feature maps of the multiple sizes, and obtaining a first feature vector corresponding to at least one target object; Obtaining correlation data between first feature vectors corresponding to the at least one target object, and labeling each target object in the at least one target object based on the correlation data; The obtaining of correlation data between the first feature vectors corresponding to the at least one target object includes obtaining the correlation data by interacting with the first feature vectors corresponding to the at least one target object; and the selecting of the first feature vectors that meet the screening condition from the first feature vectors of the feature maps of the multiple sizes includes: Obtaining association scores of the feature graphs of each size, where the association scores are used to reflect the association between the plurality of first feature vectors in the feature graph and the at least one target object; A first feature vector that meets the screening condition is selected from the first feature vectors of the feature maps of the multiple sizes based on the association scores of the feature maps of the respective sizes.

2. The method according to claim 1, characterized in that The selecting, based on the association scores of the feature graphs of each size, a first feature vector that meets the filtering condition from the first feature vectors of the feature graphs of the multiple sizes includes: determining a first quantity based on the number of first feature vectors of the feature graphs of the multiple sizes, arranging the first feature vectors of the feature graphs of the multiple sizes in order of association, and selecting the first feature vector arranged before the first quantity as the first feature vector that meets the filtering condition.

3. The method according to claim 1, characterized in that The labeling of each target object in the at least one target object based on the correlation data includes: Predicting the probability that the target object corresponds to each category based on the correlation data; The category of the target object is labeled based on the probability that the target object corresponds to each category.

4. The method according to claim 1, wherein The labeling of each target object in the at least one target object based on the correlation data includes: The correlation data is mapped to obtain a four-dimensional vector of the target object, and the position of the target object is marked based on the four-dimensional vector.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining an image annotation model corresponding to the first image, the image annotation model comprising a feature extraction module, a screening module, and a self-attention module; The acquiring feature maps of multiple sizes of the first image includes: Calling the feature extraction module to obtain feature maps of multiple sizes of the first image; The selecting a first feature vector that satisfies a screening condition from the first feature vectors of the feature graphs of the multiple sizes includes: calling the screening module to select a first feature vector that satisfies the screening condition from the first feature vectors of the feature graphs of the multiple sizes; The acquiring correlation data between the first feature vectors corresponding to the at least one target object, and labeling each target object in the at least one target object based on the correlation data, includes: The self-attention module is called to obtain correlation data between the first feature vectors corresponding to the at least one target object, and each target object in the at least one target object is labeled based on the correlation data.

6. The method according to claim 5, characterized in that The feature extraction module includes multiple layers of feature extraction submodules. The feature maps output by the feature extraction submodules in different layers have different sizes. The feature map output by the feature extraction submodule in the previous layer is the input of the feature extraction submodule in the next layer.

7. An image annotation device, characterized in that: The device comprises: An acquisition module, configured to acquire a first image, wherein the first image contains at least one target object to be labeled; The acquisition module is further configured to acquire feature maps of multiple sizes of the first image, wherein each feature map of the multiple sizes includes multiple first feature vectors; A selection module is configured to select a first feature vector that satisfies a screening condition from the first feature vectors of the feature maps of the multiple sizes to obtain a first feature vector corresponding to at least one target object, wherein the selecting the first feature vector that satisfies the screening condition from the first feature vectors of the feature maps of the multiple sizes includes: Obtaining association scores of the feature graphs of each size, where the association scores are used to reflect the association between the plurality of first feature vectors in the feature graph and the at least one target object; Selecting a first feature vector that meets the screening condition from the first feature vectors of the feature maps of the multiple sizes based on the correlation scores of the feature maps of the respective sizes; a labeling module, configured to obtain correlation data between the first feature vectors corresponding to the at least one target object, and label each target object in the at least one target object based on the correlation data; The obtaining of the correlation data between the first feature vectors corresponding to the at least one target object includes obtaining the correlation between the first feature vectors corresponding to the at least one target object by interacting with the first feature vectors corresponding to the at least one target object to obtain the correlation data.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the computer device implements the image annotation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement the image annotation method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program or instructions, and the computer program or instructions are executed by a processor to enable a computer to implement the image annotation method according to any one of claims 1 to 6.

Images