Image detection method and device, electronic equipment and storage medium

Abstract

The invention provides an image detection method and device, electronic equipment and a storage medium. During text understanding, the target preset vocabularies with the semantic similarity with the basic vocabularies larger than the preset semantic threshold value are screened from the preset vocabulary set comprising the multiple noun preset vocabularies, the similar preset vocabularies are added, and then according to the basic vocabularies and the target preset vocabularies, the semantic similarity between the basic vocabularies and the target preset vocabularies is obtained. The image with the target object is detected in the image in the video by means of the detection model, the recall rate can be improved, the detection model can not be trained or finely adjusted for the object of the specific type or the specific attribute, the object described by any type and any attribute can be detected, the cost of training the model is saved, and the detection efficiency is improved. When subsequent expansion is needed, the vocabularies needing to be expanded are added in the preset vocabulary set comprising the preset vocabularies with the noun characteristics according to requirements, training data do not need to be recollected, the detection model does not need to be retrained, and the expansion cost is low.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image detection method, device, electronic device and storage medium. Background Art

[0002] Target capture alarm based on video stream usually refers to obtaining the target object that meets specific requirements in the video through visual analysis technology, and then pushing the capture alarm to the user for further processing.

[0003] This type of technology has important economic significance and social value in the field of smart cities.

[0004] For example, by analyzing videos from public cameras, key personnel or vehicles can be monitored and specific target objects can be discovered in a timely manner, which can help public security or traffic control personnel to issue early warnings or take action, thereby improving public safety. Summary of the invention

[0005] The present application shows an image detection method, device, electronic device and storage medium.

[0006] In a first aspect, the present application provides an image detection method, the method comprising:

[0007] Obtaining a description text submitted by a user, where the description text is used to describe feature information of the target object to be detected;

[0008] Segment the description text to obtain multiple noun-like basic words in the description text;

[0009] Selecting a target preset vocabulary whose semantic similarity with the basic vocabulary is greater than a preset semantic threshold from a preset vocabulary set including a plurality of noun preset vocabulary;

[0010] Based on the basic vocabulary and the target preset vocabulary, the detection model is used to detect images with target objects in the images in the video.

[0011] In a second aspect, the present application provides an image detection device, the device comprising:

[0012] An acquisition module is used to acquire a description text submitted by a user, where the description text is used to describe feature information of a target object to be detected;

[0013] The word segmentation module is used to segment the description text to obtain multiple noun basic words in the description text;

[0014] A screening module, for screening a target preset vocabulary whose semantic similarity with the basic vocabulary is greater than a preset semantic threshold from a preset vocabulary set including a plurality of noun preset vocabulary;

[0015] The detection module is used to detect images with target objects in the images in the video based on the basic vocabulary and the target preset vocabulary with the help of the detection model.

[0016] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0017] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0018] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0019] The technical solution provided by this application may have the following beneficial effects:

[0020] In the present application, when understanding text, similar preset vocabulary is added by screening target preset vocabulary whose semantic similarity with basic vocabulary is greater than a preset semantic threshold from a preset vocabulary set including multiple noun preset vocabulary. Subsequently, based on the basic vocabulary and the target preset vocabulary, images with target objects are detected in images in the video with the help of a detection model, which can improve the recall rate, for example, increase the possibility that the detected image actually does have the target object.

[0021] The present application proposes a solution of open domain detection plus complex description text matching. It is not necessary to train and fine-tune the detection model for objects of specific types or specific attributes, and it is possible to detect objects of any category and with any attribute description, thus saving the cost of training the model. When expansion is needed in the future, the vocabulary that needs to be expanded can be added to the preset vocabulary set including multiple noun preset vocabularies as needed. There is no need to re-collect training data or retrain the detection model, and the expansion cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of the steps of an image detection method of the present application.

[0023] Figure 2 It is a schematic diagram of a method for screening target preset vocabulary in the present application.

[0024] Figure 3 It is a schematic diagram of a method for obtaining feature similarity in the present application.

[0025] Figure 4 It is a flowchart of the steps of an image detection method of the present application.

[0026] Figure 5 It is a schematic diagram of a method for screening candidate images in the present application.

[0027] Figure 6 It is a schematic diagram of a method for acquiring an image with a target object in the present application.

[0028] Figure 7 It is a structural block diagram of an image detection device of the present application.

[0029] Figure 8 It is a block diagram of an electronic device of the present application.

[0030] Fig. 9 It is a block diagram of an electronic device of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] Current target object snapshot alerts are mostly based on limited categories and attributes, such as humans or vehicles, such as the license plate number or color of a vehicle.

[0033] Currently, most capture algorithm models for target objects with limited categories and attributes detect images in videos, and the detected images contain target objects with limited categories and attributes.

[0034] However, if new categories or attributes need to be introduced later so that the algorithm model can detect target objects of the new categories or attributes, it is often necessary to add new training data for the new categories or attributes and iterate the algorithm model using the training data for the new categories or attributes before detection of objects of the new categories or attributes can be introduced. However, this method is costly and has poor scalability.

[0035] Secondly, existing algorithm models (such as open domain detection algorithm models, etc.) do not fully understand complex descriptions and it is difficult to comprehensively and accurately detect target objects with arbitrary complex attribute descriptions in videos.

[0036] To this end, the solution of this application is proposed.

[0037] Before introducing the solution of this application, the technical terms that may be involved in this application are first explained.

[0038] Grounding DINO is a new SOTA (state-of-the-art) zero-shot object detection model. It aims to fuse the designs of DINO and GLIP (Grounded Language-Image Pre-training). By combining the Transformer-based detector DINO with localization pre-training, it can detect arbitrary objects, such as manually entered category names or referential expressions. This model can recognize and distinguish a variety of objects, even those that have not appeared in the training set, so it has high flexibility and generalization ability. In terms of specific implementation, the Grounding DINO model adopts a combination of text encoder and image encoder, and achieves accurate detection of objects in images through mechanisms such as cross-modal feature fusion and query selection guided by text semantics. In addition, the model also uses components such as feature enhancer and multimodal decoder to further improve detection performance.

[0039] The K-means clustering algorithm is a commonly used unsupervised learning algorithm that aims to divide a data set into K different clusters so that the data points in the same cluster are as similar as possible, while the data points between different clusters are as different as possible.

[0040] Reference Figure 1 , shows a flowchart of the steps of an image detection method of the present application, wherein the method comprises:

[0041] In step S101, a description text submitted by a user is obtained, where the description text is used to describe feature information of a target object to be detected.

[0042] In the present application, sometimes the user needs to filter out the target object in the video. Filtering out the target object in the video can be understood as filtering out images with the target object in the video. In this case, the user can summarize the feature information of the target object and draft a description text that can describe the feature information of the target object, and then submit the description text to the electronic device so that the electronic device obtains the description text submitted by the user, and then executes step S102.

[0043] For example, when a user needs to filter out the target object "a man in black clothes dragging a suitcase" in a video, the target object "a man in black clothes dragging a suitcase" has the features of "man", "wearing black clothes" and "dragging a suitcase".

[0044] In this example, “a man in black clothes dragging a suitcase” may be the description text.

[0045] In step S102, the description text is segmented to obtain a plurality of noun basic words in the description text.

[0046] In the present application, any segmentation method currently available on the market can be used to segment the description text to obtain multiple words in the description text, and multiple noun-like basic words can be determined from the multiple words obtained.

[0047] For example, the description text "a man in black clothes dragging a suitcase" is segmented to obtain the words "wear", "black", "clothes", "drag", "suitcase", "of", and "man".

[0048] The noun words include: "clothes", "suitcase" and "man".

[0049] Thus, the basic noun words are: "clothes", "suitcase" and "man".

[0050] In step S103, a target preset vocabulary whose semantic similarity with the basic vocabulary is greater than a preset semantic threshold is selected from the preset vocabulary set including a plurality of noun preset vocabulary.

[0051] In one embodiment of the present application, the lexical features of the basic vocabulary may be acquired based on a CLIP (Contrastive Language-Image Pre-Training) model.

[0052] For any preset word in the preset word set, the lexical features of the preset word can be obtained based on the CLIP model.

[0053] The vocabulary feature of the basic vocabulary may be a feature vector, and the vocabulary feature of the preset vocabulary may be a feature vector.

[0054] The dimensions of the feature vectors of the basic vocabulary and the feature vectors of each preset vocabulary are the same, for example, the dimensions are 1*N, representing 1 row and N columns, N is the number of elements in the vector, and the specific value of N can be determined according to actual conditions, and this application does not limit this.

[0055] Then, the feature similarity between the lexical features of the basic vocabulary and the lexical features of the preset vocabulary may be obtained.

[0056] For example, the cosine similarity between the feature vector of the basic vocabulary and the feature vector of the preset vocabulary may be calculated and used as the feature similarity between the vocabulary feature of the basic vocabulary and the vocabulary feature of the preset vocabulary.

[0057] Then, the semantic similarity between the basic vocabulary and the preset vocabulary can be obtained according to the feature similarity. For example, the feature similarity between the lexical features of the basic vocabulary and the lexical features of the preset vocabulary is determined as the semantic similarity between the basic vocabulary and the preset vocabulary.

[0058] When the semantic similarity is greater than a preset semantic threshold, the preset vocabulary may be determined as a target preset vocabulary.

[0059] Alternatively, when the semantic similarity is less than or equal to a preset semantic threshold, the preset vocabulary is not determined as the target preset vocabulary.

[0060] The specific value of the preset semantic threshold can be determined according to actual conditions, and this application does not limit this and will not be described in detail here.

[0061] Figure 2 It is a schematic diagram of a method for selecting target preset vocabulary whose semantic similarity with basic vocabulary is greater than a preset semantic threshold in a preset vocabulary set including multiple noun preset vocabulary. Both the basic vocabulary and the target preset vocabulary can be input into a detection model.

[0062] Figure 2 The v in it stands for verb, adj stands for adjective, n stands for noun, and aux stands for auxiliary word.

[0063] Figure 3 It is a schematic diagram of a method for obtaining feature similarity between lexical features of basic vocabulary and lexical features of preset vocabulary in a preset vocabulary set.

[0064] The preset vocabulary set is set in advance by the technician, and the multiple noun preset vocabulary in the preset vocabulary set is also counted and set in advance by the technician.

[0065] In one example, the plurality of noun preset words in the preset word set include: coat, car, bicycle, glasses, helmet, pants, box, dog, cat, pet, suitcase, electric car, motorcycle, table, potted plant, and woman, etc. The above examples are only illustrative examples and are not intended to limit the scope of protection of the present application.

[0066] The present application does not limit the number of the multiple noun preset words in the preset vocabulary set. For example, it can be 2000 words, 3000 words, or even more words.

[0067] After calculation of semantic similarity, in a preset vocabulary set including multiple noun preset vocabulary, the target preset vocabulary screened out whose semantic similarity with the basic vocabulary "clothes", "suitcase" and "man" is greater than a preset semantic threshold may include: "top", "pants", "box", "suitcase" and "woman".

[0068] There may be overlaps between the basic vocabulary and the preset vocabulary in the preset vocabulary set. The target preset vocabulary and the basic vocabulary are different vocabulary (ie, they do not overlap).

[0069] In step S104, images with target objects are detected in the images in the video using a detection model based on the basic vocabulary and the target preset vocabulary.

[0070] In one embodiment, according to the detection model, the basic vocabulary and the target preset vocabulary can be used as prompt words to detect images with target objects in the images in the video.

[0071] The detection model may include: an open domain detection model, and the open domain detection model may include: a grounding dino model / algorithm, etc.

[0072] For example, basic vocabulary and target preset vocabulary can be input into the detection model as prompt words, and a video can be input into the detection model so that the detection model uses the basic vocabulary and target preset vocabulary as prompt words to detect images with target objects in the images in the video.

[0073] Then, the bounding rectangle of the target object can be marked in the image with the target object, and the image with the target object can be output. For example, the image with the target object can be stored for later retrieval, and the image with the target object can be sent to a client used by the user for viewing by the user.

[0074] In the present application, when understanding text, similar preset vocabulary is added by screening target preset vocabulary whose semantic similarity with basic vocabulary is greater than a preset semantic threshold from a preset vocabulary set including multiple noun preset vocabulary. Subsequently, based on the basic vocabulary and the target preset vocabulary, images with target objects are detected in images in the video with the help of a detection model, which can improve the recall rate, for example, increase the possibility that the detected image actually does have the target object.

[0075] The present application proposes a solution of open domain detection plus complex description text matching. It is not necessary to train and fine-tune the detection model for objects of specific types or specific attributes, and it is possible to detect objects of any category and with any attribute description, thus saving the cost of training the model. When expansion is needed in the future, the vocabulary that needs to be expanded can be added to the preset vocabulary set including multiple noun preset vocabularies as needed. There is no need to re-collect training data or retrain the detection model, and the expansion cost is low.

[0076] In one embodiment of the present application, see Figure 4 , step S104 includes:

[0077] In step S201, the classification label of each object in each image in the video is detected according to the detection model.

[0078] The video is input into the detection model so that the detection model detects the classification label of each object in each image in the video.

[0079] In one embodiment, in most cases, the classification label of each object in each image in the detected video is often located in a preset vocabulary set including a plurality of noun preset vocabularies.

[0080] Objects can include people, animals, trees, buildings, trash cans, etc.

[0081] In step S202, among the objects in the images in the video, relevant objects whose classification labels are related to the basic vocabulary and / or the target preset vocabulary are screened.

[0082] In the present application, for any one of the basic vocabulary and the target preset vocabulary, in each image in the video, the objects whose classification label is the vocabulary are screened and used as related objects whose classification labels are related to the vocabulary, that is, related objects whose classification labels are related to the basic vocabulary and / or the target preset vocabulary, and the same is true for each other word in the basic vocabulary and the target preset vocabulary.

[0083] In step S203, among the images in the video, images containing relevant objects are screened.

[0084] In this application, this step can be implemented through the following process, including:

[0085] 2031. In each image in the video, screen multiple candidate images where relevant objects are located.

[0086] In the present application, any image having a related object can be used as a candidate image.

[0087] In one embodiment of the present application, related objects can be tracked in each image in a video. For example, a multi-target tracking algorithm such as bytetrack can be used to match objects in adjacent images. For example, related objects can be paired, and multiple candidate images of the same related object can be organized into an image sequence according to the sequence of images. Each candidate image in the image sequence has a trackid (track identity document), and different candidate images in the image sequence have different trackids.

[0088] 2032. Screen at least two candidate images from among the multiple candidate images, wherein posture features of related objects in at least the two candidate images are different.

[0089] In one embodiment, the local images where the relevant objects are located in each candidate image may be acquired respectively, and the sizes of the local images may be the same.

[0090] A partial image is an image of a part of the candidate image in which it is located. The partial image has all the contents of the relevant object in the candidate image in which it is located, that is, the partial image has the contents of the relevant object that the candidate image in which the partial image is located has.

[0091] Based on the CLIP model, local image features of each local image are obtained respectively.

[0092] The local image features of each local image are classified to obtain at least two feature sets, one feature set having the local image features of at least one local image, and the local image features in different feature sets do not belong to the same category.

[0093] For example, based on the K-means clustering algorithm, the local image features of each local image may be classified to obtain at least two feature sets.

[0094] The target object is affected by the shooting angle, shape, posture, light occlusion and other environmental factors in the entire trajectory. If a candidate image is selected from multiple candidate images, the relevant information of the target object may be lost. For example, the relevant information of the target object in other candidate images is lost, which may cause the final detection result to be inaccurate and unstable.

[0095] This embodiment adopts a video stream solution to detect, track, capture, and push images of the target object. By integrating the information of the video stream, relevant information of different postures of the target object in at least two candidate images can be retained, so that more accurate results can be obtained.

[0096] For example, in the present application, the images where the screened related objects are located are multiple images, such as at least two candidate images, and the posture features of the related objects in at least two candidate images are different, so as to ensure the diversity / diversity of the features of the related objects in at least two candidate images, taking into account the performance of different characteristics of the related objects in the entire trajectory, rather than focusing on a certain characteristic, thereby improving the recall rate.

[0097] Among them, the K-means clustering algorithm can be used to obtain images of representative features of the relevant objects in all candidate images where the relevant objects are located, thereby avoiding repeated push of images with the same characteristics.

[0098] For example, for any feature set, the image quality of the local images corresponding to each local image feature in the feature set can be obtained. Then, among multiple candidate images, the candidate image corresponding to the local image with the highest image quality is screened to obtain the candidate image corresponding to the feature set. That is, among the candidate images where the local images corresponding to each local image feature in the feature set are located, the candidate image corresponding to the local image with the highest image quality is screened to obtain the candidate image corresponding to the feature set. The same is true for each other feature set, thereby obtaining the candidate images corresponding to each feature set.

[0099] In one embodiment, when obtaining the image quality of the local images corresponding to the local image features in the feature set, for the local image corresponding to any local image feature in the feature set, the size of the circumscribed rectangular frame of the relevant object in the local image corresponding to the local image feature can be obtained. The image quality of the local image corresponding to the local image feature is obtained based on the size of the circumscribed rectangular frame of the relevant object in the local image corresponding to the local image feature.

[0100] For example, if the size of the circumscribed rectangular frame of the relevant object in the partial image is larger, the image quality of the partial image is greater, or if the size of the circumscribed rectangular frame of the relevant object in the partial image is smaller, the image quality of the partial image is smaller.

[0101] The local images can be sorted in descending order according to the size of the circumscribed rectangular frame of the relevant object in the local image corresponding to each local image feature, and the image quality of the local image ranked first is the highest, and the image quality of the local image ranked last is the lowest.

[0102] In one embodiment, when obtaining the image quality of the local images corresponding to the local image features in the feature set, for the local image corresponding to any local image feature in the feature set, the blocked area of ​​the relevant object in the local image corresponding to the local image feature can be obtained. The image quality of the local image corresponding to the local image feature is obtained based on the blocked area of ​​the relevant object in the local image corresponding to the local image feature.

[0103] For example, if the blocked area of ​​the relevant object in the partial image is larger, the image quality of the partial image is smaller; or, if the blocked area of ​​the relevant object in the partial image is smaller, the image quality of the partial image is larger.

[0104] The partial images may be sorted in order of the blocked areas of the relevant objects in the partial images from small to large, the first-ranked partial image having the highest image quality and the last-ranked partial image having the lowest image quality.

[0105] In one embodiment, when obtaining the image quality of the local images corresponding to each local image feature in the feature set, for the local image corresponding to any local image feature in the feature set, the completeness of the related object in the local image corresponding to the local image feature can be obtained. Based on the completeness of the related object in the local image corresponding to the local image feature, the image quality of the local image corresponding to the local image feature is obtained.

[0106] For example, if the integrity of the relevant object in the partial image is greater, the image quality of the partial image is greater, or if the integrity of the relevant object in the partial image is smaller, the image quality of the partial image is smaller.

[0107] The partial images may be sorted in descending order according to the completeness of the related objects in the partial images corresponding to the partial image features. The partial image ranked first has the highest image quality, and the partial image ranked last has the lowest image quality.

[0108] The completeness can be understood as whether there are any missing parts of the relevant object in the partial image, and the missing parts can be understood as: the content that the relevant object itself has but the content is not reflected in the partial image.

[0109] Through the above embodiments, the quality of the image where the related object is obtained can be guaranteed to be higher, and the features of the related object in the image where the related object is located are more prominent, which facilitates the subsequent acquisition of an image with the target object based on the image where the related object is located.

[0110] Figure 5 A schematic diagram of a method for screening at least two candidate images from a plurality of candidate images is shown.

[0111] Figure 5 In the clustered categories, 3 candidate images are candidate images of category 1, 5 candidate images are candidate images of category 2, and 2 candidate images are candidate images of category 3. Select a candidate image (push) from the 3 candidate images of category 1. Figure 1 ), select a candidate image from the 5 candidate images of category 2 (push Figure 2 ), select a candidate image from the two candidate images of category 3 (push Figure 3 ), and finally get the Figure 1 ,push Figure 2 And push Figure 3 , a total of 3 candidate images.

[0112] In step S204, an image having the target object is acquired according to the image where the related object is located.

[0113] In one embodiment, the image where the related object is located may be determined as the image having the target object.

[0114] However, considering that the characteristics of some objects are difficult to describe clearly even with descriptive texts, and they lack intuitiveness for users, it is supported that users can submit a descriptive image corresponding to the descriptive text when submitting the descriptive text of the target object, so as to serve as preset information to assist the electronic device in obtaining an image with the target object based on the image where the related object is located.

[0115] Thus, in another embodiment of the present application, this step can be implemented by the following process, including:

[0116] 2041. Obtain a description image corresponding to the description text submitted by the user, wherein the description image has a target object and / or an object similar to the target object, and an image similarity between the similar object and the target object is greater than a preset image threshold.

[0117] The preset image threshold is set by technicians based on actual conditions, and the specific value of the preset image threshold in this application is not limited.

[0118] The image similarity between the similar object and the target object may include: cosine similarity between a feature vector of the similar object and a feature vector of the target object, etc.

[0119] 2042. For any one of the images where the relevant object is located, verify whether the image contains the target object at least according to the description image. If the image contains the target object, determine the image as an image containing the target object. Alternatively, if the image does not contain the target object, do not determine the image as an image containing the target object. The same is true for each of the other images where the relevant object is located. Thus, all images containing the target object are obtained.

[0120] Wherein, when verifying whether the image contains the target object at least according to the description image, it can be implemented through the following process, including:

[0121] 21) Obtain image features describing the image based on the CLIP model, and obtain image features of the image based on the CLIP model.

[0122] 22) Verify whether the image contains a target object based on at least image features describing the image and image features of the image.

[0123] For example, in one embodiment, the feature similarity between the image feature describing the image and the image feature of the image may be obtained. When the feature similarity is greater than a preset feature threshold, it is determined that the image contains the target object. Alternatively, when the feature similarity is less than or equal to the preset feature threshold, it is determined that the image does not contain the target object.

[0124] In one embodiment, there are multiple description images corresponding to the description text submitted by the user (considering the use of multiple description images to reflect the appearance of different scenes from different angles of the target object, etc.), and the image features of each description image can be obtained separately based on the CLIP model, and then the image features of each description image are fused (for example, calculating the average value of the feature vector, etc.), and the image features obtained after the fusion can be used as the image features of the description image in this step 22).

[0125] The image features describing the image may be feature vectors, and the image features of the image may be feature vectors.

[0126] The dimensions of the feature vector describing the image and the feature vector of the image are the same, for example, both are 1*N dimensions, representing 1 row and N columns, N is the number of elements in the vector, and the specific value of N can be determined according to actual conditions, and this application does not limit this.

[0127] In this way, when obtaining the feature similarity between the image feature describing the image and the image feature of the image, the cosine similarity between the feature vector describing the image and the feature vector of the image can be calculated and used as the feature similarity between the image feature describing the image and the image feature of the image.

[0128] Alternatively, in another embodiment of the present application, the lexical features of the basic vocabulary may be obtained based on the CLIP model, and the image features describing the image are fused with the lexical features of the basic vocabulary to obtain fused features.

[0129] The vocabulary features of the basic vocabulary may be feature vectors, and the image features describing the image may be feature vectors.

[0130] The dimensions of the feature vector of the basic vocabulary and the feature vector describing the image are the same, for example, both are 1*N dimensions, representing 1 row and N columns, where N is the number of elements in the vector. The specific value of N can be determined according to actual conditions, and this application does not limit this.

[0131] The fusion of the image features describing the image and the lexical features of the basic vocabulary can be understood as: adding or calculating the average of the elements in the same position in the feature vector of the basic vocabulary and the feature vector describing the image to obtain another 1*N feature vector (i.e., the fused feature).

[0132] Then, the feature similarity between the fused features and the image features of the image can be obtained.

[0133] For example, the cosine similarity between the feature vector of the fused feature and the feature vector of the image may be calculated and used as the feature similarity between the fused feature and the image feature of the image.

[0134] When the feature similarity is greater than the preset feature threshold, it can be determined that the image contains the target object. Alternatively, when the feature similarity is less than or equal to the preset feature threshold, it can be determined that the image does not contain the target object.

[0135] In the above two embodiments, the specific value of the preset characteristic threshold value can be determined according to actual conditions, and this application does not impose any limitation on this.

[0136] Figure 6 A schematic diagram of a method for acquiring an image having a target object according to an image in which a related object is located is shown.

[0137] The present application verifies whether the target object is actually present in the image where the relevant object is located through the description image corresponding to the description text submitted by the user, so as to filter out the images that do not actually have the target object in the detection results (the images where the relevant object is located) to avoid erroneous output, thereby achieving both accuracy and recall rate.

[0138] In addition, when verifying whether the target object actually exists in the image where the relevant object is located, not only text features but also more intuitive and clear image features are taken into account, which reduces the ambiguity of the features and improves the accuracy of verification.

[0139] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0140] Reference Figure 7 , shows an image detection device of the present application, the device comprising:

[0141] The acquisition module 11 is used to acquire the description text submitted by the user, where the description text is used to describe the feature information of the target object to be detected;

[0142] A word segmentation module 12 is used to segment the description text to obtain a plurality of noun basic words in the description text;

[0143] A screening module 13 is used to screen a target preset vocabulary whose semantic similarity with the basic vocabulary is greater than a preset semantic threshold from the preset vocabulary set including a plurality of noun preset vocabulary;

[0144] The detection module 14 is used to detect images with target objects in the images in the video based on the basic vocabulary and the target preset vocabulary with the help of the detection model.

[0145] In an optional implementation, the screening module includes:

[0146] A first acquisition unit, configured to acquire lexical features of basic vocabulary based on a contrastive language-image pre-trained CLIP model;

[0147] The second acquisition unit is used to acquire the lexical features of any preset vocabulary in the preset vocabulary set based on the CLIP model; the third acquisition unit is used to acquire the feature similarity between the lexical features of the basic vocabulary and the lexical features of the preset vocabulary; the fourth acquisition unit is used to acquire the semantic similarity between the basic vocabulary and the preset vocabulary based on the feature similarity; the determination unit is used to determine the preset vocabulary as the target preset vocabulary when the semantic similarity is greater than a preset semantic threshold.

[0148] In an optional implementation, the detection module includes:

[0149] A detection unit, used to detect the classification label of each object in each image in the video according to the detection model;

[0150] A first screening unit is used to screen out relevant objects whose classification labels are related to the basic vocabulary and / or the target preset vocabulary from among the objects in the images in the video;

[0151] A second screening unit is used to screen images containing relevant objects from among the images in the video;

[0152] The fifth acquisition unit is used to acquire an image having the target object according to the image where the related object is located.

[0153] In an optional implementation, the second screening unit includes:

[0154] A first screening subunit is used to screen multiple candidate images where relevant objects are located from among the images in the video;

[0155] The second screening subunit is used to screen at least two candidate images from a plurality of candidate images, wherein the posture features of the related objects in the at least two candidate images are different.

[0156] In an optional implementation, the first screening subunit is specifically used to: respectively obtain the local images where the relevant objects are located in each candidate image; respectively obtain the local image features of each local image based on the CLIP model; classify the local image features of each local image to obtain at least two feature sets, one feature set has local image features of at least one local image, and the local image features in different feature sets do not belong to the same category; for any feature set, obtain the image quality of the local images corresponding to each local image feature in the feature set, and among multiple candidate images, screen the candidate image corresponding to the local image with the largest image quality to obtain the candidate image corresponding to the feature set.

[0157] In an optional implementation, the first screening sub-unit is specifically used to obtain, for a local image corresponding to any local image feature in the feature set, the size of a circumscribed rectangular frame of a related object in the local image corresponding to the local image feature; and obtain the image quality of the local image corresponding to the local image feature based on the size of the circumscribed rectangular frame of the related object in the local image corresponding to the local image feature.

[0158] In an optional implementation, the first screening sub-unit is specifically used to obtain, for a local image corresponding to any local image feature in the feature set, an occluded area of ​​a related object in the local image corresponding to the local image feature; and obtain the image quality of the local image corresponding to the local image feature based on the occluded area of ​​the related object in the local image corresponding to the local image feature.

[0159] In an optional implementation, the first screening subunit is specifically used to obtain, for a local image corresponding to any local image feature in the feature set, the completeness of a related object in the local image corresponding to the local image feature; and obtain the image quality of the local image corresponding to the local image feature based on the completeness of the related object in the local image corresponding to the local image feature.

[0160] In an optional implementation, the fifth acquiring unit includes:

[0161] an acquisition subunit, configured to acquire a description image corresponding to the description text and submitted by a user, wherein the description image has the target object and / or a similar object to the target object, and the image similarity between the similar object and the target object is greater than a preset image threshold;

[0162] The verification subunit is used to verify whether any one of the images where the relevant object is located has the target object based on at least the description image, and if the image has the target object, determine the image as an image with the target object.

[0163] In an optional implementation, the verification subunit is specifically used to: obtain image features describing the image based on the CLIP model; obtain image features of the image based on the CLIP model; and verify whether there is a target object in the image based on at least the image features describing the image and the image features of the image.

[0164] In an optional implementation, the verification subunit is specifically used to: obtain feature similarity between the image features describing the image and the image features of the image; when the feature similarity is greater than a preset feature threshold, determine that there is a target object in the image; or, when the feature similarity is less than or equal to the preset feature threshold, determine that there is no target object in the image.

[0165] In an optional implementation, the verification subunit is specifically used to: obtain lexical features of basic vocabulary based on the CLIP model; fuse the image features describing the image with the lexical features of the basic vocabulary to obtain fused features; obtain feature similarity between the fused features and the image features of the image; when the feature similarity is greater than a preset feature threshold, determine that the image contains a target object; or, when the feature similarity is less than or equal to the preset feature threshold, determine that the image does not contain a target object.

[0166] In the present application, when understanding text, similar preset vocabulary is added by screening target preset vocabulary whose semantic similarity with basic vocabulary is greater than a preset semantic threshold from a preset vocabulary set including multiple noun preset vocabulary. Subsequently, based on the basic vocabulary and the target preset vocabulary, images with target objects are detected in images in the video with the help of a detection model, which can improve the recall rate, for example, increase the possibility that the detected image actually does have the target object.

[0167] The present application proposes a solution of open domain detection plus complex description text matching. It is not necessary to train and fine-tune the detection model for objects of specific types or specific attributes, and it is possible to detect objects of any category and with any attribute description, thus saving the cost of training the model. When expansion is needed in the future, the vocabulary that needs to be expanded can be added to the preset vocabulary set including multiple noun preset vocabularies as needed. There is no need to re-collect training data or retrain the detection model, and the expansion cost is low.

[0168] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0169] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0170] Figure 8 800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0171] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0172] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0173] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0174] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0175] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also monitor the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0176] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0177] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0178] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can monitor the open / closed state of the device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also monitor the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to monitor the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0179] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0180] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0181] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0182] Fig. 91 is a block diagram of an electronic device 1900 shown in the present application. For example, the electronic device 1900 may be provided as a server.

[0183] Reference Fig. 9 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0184] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0185] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0187] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0188] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0189] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0193] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0194] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image detection method, characterized in that: The method comprises: Obtaining a description text submitted by a user, where the description text is used to describe feature information of the target object to be detected; Segment the description text to obtain multiple noun-like basic words in the description text; Selecting a target preset vocabulary whose semantic similarity with the basic vocabulary is greater than a preset semantic threshold from a preset vocabulary set including a plurality of noun preset vocabulary; Based on the basic vocabulary and the target preset vocabulary, the detection model is used to detect images with target objects in the images in the video.

2. The method according to claim 1, characterized in that The step of selecting a target preset vocabulary having a semantic similarity with a basic vocabulary greater than a preset semantic threshold from a preset vocabulary set including a plurality of noun preset vocabulary comprises: Obtaining lexical features of basic vocabulary based on contrastive language-image pre-training CLIP model; For any preset vocabulary in the preset vocabulary set, the lexical features of the preset vocabulary are obtained based on the CLIP model; the feature similarity between the lexical features of the basic vocabulary and the lexical features of the preset vocabulary is obtained; the semantic similarity between the basic vocabulary and the preset vocabulary is obtained according to the feature similarity; when the semantic similarity is greater than a preset semantic threshold, the preset vocabulary is determined as a target preset vocabulary.

3. The method according to claim 1, characterized in that The method of detecting an image having a target object in an image in a video by using a detection model according to the basic vocabulary and the target preset vocabulary includes: According to the detection model, the classification label of each object in each image in the video is detected; Among the objects in the images in the video, select the relevant objects whose classification labels are related to the basic vocabulary and / or the target preset vocabulary; Among the images in the video, select the images where the relevant objects are located; Get the image with the target object according to the image where the related object is located.

4. The method according to claim 3, characterized in that The step of screening the images containing the relevant objects from among the images in the video includes: In each image in the video, multiple candidate images where relevant objects are located are screened; At least two candidate images are screened from among the multiple candidate images, and the posture features of the related objects in the at least two candidate images are different.

5. The method according to claim 4, characterized in that The step of screening at least two candidate images from a plurality of candidate images comprises: Respectively obtaining the local images where the relevant objects are located in each candidate image; Based on the CLIP model, local image features of each local image are obtained respectively; Classifying the local image features of each local image to obtain at least two feature sets, one feature set having the local image features of at least one local image, and the local image features in different feature sets do not belong to the same class; For any feature set, the image quality of the local images corresponding to each local image feature in the feature set is obtained, and among multiple candidate images, the candidate image corresponding to the local image with the largest image quality is screened to obtain the candidate image corresponding to the feature set.

6. The method according to claim 5, characterized in that The obtaining the image quality of the local images respectively corresponding to the local image features in the feature set includes: For a local image corresponding to any local image feature in the feature set, obtaining the size of a circumscribed rectangular frame of a related object in the local image corresponding to the local image feature; The image quality of the local image corresponding to the local image feature is obtained according to the size of the circumscribed rectangular frame of the relevant object in the local image corresponding to the local image feature.

7. The method according to claim 5, characterized in that The obtaining the image quality of the local images respectively corresponding to the local image features in the feature set includes: For a local image corresponding to any local image feature in the feature set, obtaining an obscured area of ​​a related object in the local image corresponding to the local image feature; The image quality of the local image corresponding to the local image feature is obtained according to the blocked area of ​​the relevant object in the local image corresponding to the local image feature.

8. The method according to claim 5, characterized in that The obtaining the image quality of the local images respectively corresponding to the local image features in the feature set includes: For a local image corresponding to any local image feature in the feature set, obtaining the completeness of a related object in the local image corresponding to the local image feature; The image quality of the local image corresponding to the local image feature is obtained according to the completeness of the related object in the local image corresponding to the local image feature.

9. The method according to claim 3, characterized in that: The step of acquiring an image having a target object according to an image where a related object is located includes: Acquire a description image corresponding to the description text and submitted by a user, wherein the description image has the target object and / or a similar object to the target object, and an image similarity between the similar object and the target object is greater than a preset image threshold; For any one of the images where the relevant object is located, whether the image contains the target object is verified at least according to the description image, and if the image contains the target object, the image is determined to be an image containing the target object.

10. The method according to claim 9, characterized in that The step of verifying whether the image contains the target object at least according to the description image comprises: Obtaining image features describing the image based on the CLIP model; Obtaining image features of the image based on the CLIP model; Whether the image contains a target object is verified at least based on image features describing the image and image features of the image.

11. The method according to claim 10, characterized in that The step of verifying whether the image contains a target object at least based on the image feature describing the image and the image feature of the image comprises: Obtaining feature similarity between the image feature describing the image and the image feature of the image; When the feature similarity is greater than a preset feature threshold, determining that the image contains a target object; or, When the feature similarity is less than or equal to the preset feature threshold, it is determined that there is no target object in the image.

12. The method according to claim 10, characterized in that The step of verifying whether the image contains a target object at least based on the image feature describing the image and the image feature of the image comprises: Obtaining lexical features of basic vocabulary based on the CLIP model; Fusing the image features describing the image with the vocabulary features of the basic vocabulary to obtain fused features; Obtaining feature similarity between the fusion feature and the image feature of the image; When the feature similarity is greater than a preset feature threshold, determining that the image contains a target object; or, When the feature similarity is less than or equal to the preset feature threshold, it is determined that there is no target object in the image.

13. An image detection device, characterized in that: The device comprises: An acquisition module is used to acquire a description text submitted by a user, where the description text is used to describe feature information of a target object to be detected; The word segmentation module is used to segment the description text to obtain multiple noun basic words in the description text; A screening module, for screening a target preset vocabulary whose semantic similarity with the basic vocabulary is greater than a preset semantic threshold from a preset vocabulary set including a plurality of noun preset vocabulary; The detection module is used to detect images with target objects in the images in the video based on the basic vocabulary and the target preset vocabulary with the help of the detection model.

14. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by the processor.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images