An image retrieval method, device and electronic equipment

Abstract

The application discloses an image retrieval method and device and electronic equipment. In the application, target text feature information of target text information is determined based on an image retrieval model, and a plurality of image-text group feature information is spliced according to the target text feature information and image feature information of a plurality of images in an image library. The image retrieval model is used to determine retrieval category information corresponding to the plurality of image-text group feature information, and target retrieval images are determined according to images corresponding to the same retrieval category information as target category information carried in the target text information. Compared with the manual review and retrieval scheme, the application greatly reduces manpower and material resources, improves image retrieval efficiency, avoids false positives or missed detection caused by manual subjective retrieval, and improves the accuracy of image retrieval. The technical scheme provided by the application has strong robustness, reliability, operability and interpretability, and meets the trustworthy characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an image retrieval method and device and electronic equipment. BACKGROUND

[0002] In a traffic scene, a video monitoring system will capture many illegal scenes and save them as images. Illegal scenes include, for example, illegal parking, running a red light, not giving way to pedestrians, illegal lane changing, unlicensed vehicles, illegal vehicles entering a highway, and occupying an emergency lane.

[0003] At present, a large number of images formed by a large number of captured events are saved together. When it is necessary to retrieve images of a certain type of illegal scene, a large number of images saved need to be checked and classified one by one by a large number of people, and the retrieved images are finally selected by the people. The problem with the prior art is that, due to the large number of captured images, the difficulty of auditing is high, which leads to the need for a large amount of manpower and material resources for the auditing and retrieval of traffic illegal scenes, which is very time-consuming and labor-intensive, and the retrieval efficiency is very low. In addition, manual subjective retrieval is prone to false positives or false negatives, resulting in poor accuracy. SUMMARY

[0004] The present application provides an image retrieval method, device and electronic equipment to solve the problem of time-consuming and labor-intensive image retrieval and poor accuracy in the prior art.

[0005] In a first aspect, the present application provides an image retrieval method, which comprises:

[0006] obtaining target text information of an image to be retrieved, inputting the target text information into an image retrieval model, and determining a target retrieval image;

[0007] The image retrieval model is used to determine target text feature information of the target text information, obtain image feature information of a plurality of images in an image library, respectively splice the target text feature information and the image feature information of the plurality of images to obtain a plurality of image-text group feature information, and respectively determine retrieval category information corresponding to the plurality of image-text group feature information.

[0008] Selecting image-text group feature information corresponding to retrieval category information identical to target category information carried in the target text information, and determining a target retrieval image according to an image corresponding to the image-text group feature information.

[0009] In a second aspect, the present application provides an image retrieval device, which comprises:

[0010] The first determining module is configured to acquire target text information of an image to be searched, input the target text information into an image search model, and determine a target search image; wherein the image search model is configured to determine target text feature information of the target text information; acquire image feature information of a plurality of images in an image library, splice the target text feature information and the image feature information of the plurality of images respectively, and obtain a plurality of image-text group feature information; and determine search category information corresponding to the plurality of image-text group feature information respectively.

[0011] The second determining module is configured to select image-text group feature information corresponding to search category information identical to target category information carried in the target text information, and determine a target search image according to an image corresponding to the image-text group feature information.

[0012] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0013] The memory is configured to store a computer program.

[0014] The processor is configured to execute the program stored on the memory, and implement the method steps.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method steps.

[0016] The present application provides an image search method, device and electronic device, and the method comprises the following steps: acquiring target text information of an image to be searched, inputting the target text information into an image search model, and determining a target search image; wherein the image search model is configured to determine target text feature information of the target text information; acquiring image feature information of a plurality of images in an image library, splicing the target text feature information and the image feature information of the plurality of images respectively, and obtaining a plurality of image-text group feature information; determining search category information corresponding to the plurality of image-text group feature information respectively; selecting image-text group feature information corresponding to search category information identical to target category information carried in the target text information, and determining a target search image according to an image corresponding to the image-text group feature information.

[0017] The above technical solution has the following advantages or beneficial effects:

[0018] In the present application, the target text feature information of the target text information is determined based on the image retrieval model, and the image feature information of a plurality of images in the image library is obtained based on the target text feature information and the image feature information of the plurality of images in the image library. A plurality of image-text group feature information is spliced, and the retrieval category information corresponding to the plurality of image-text group feature information is determined based on the image retrieval model. The target retrieval image is determined based on the image corresponding to the retrieval category information same as the target category information carried in the target text information. Compared with the manual auditing and retrieval scheme, the human and material resources are greatly reduced, the image retrieval efficiency is improved, and the problem of poor accuracy caused by manual subjective retrieval is avoided. The accuracy of image retrieval is improved. The technical scheme provided in the present application has strong robustness, reliability, operability and interpretability, and meets the reliable characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 The image retrieval process schematic diagram provided in the present application is provided.

[0021] Figure 2 The process schematic diagram for determining the retrieval category information corresponding to the plurality of image-text group feature information provided in the present application is provided.

[0022] Figure 3 The process schematic diagram for obtaining the image feature information of a plurality of images in the image library provided in the present application is provided.

[0023] Figure 4 The process schematic diagram for determining the plurality of image-text group feature information provided in the present application is provided.

[0024] Figure 5 The training process schematic diagram of the feature bidirectional matching model provided in the present application is provided.

[0025] Figure 6 The matching image-text group schematic diagram provided in the present application is provided.

[0026] Figure 7 The training process schematic diagram of the image retrieval model provided in the present application is provided.

[0027] Figure 8 The process schematic diagram for determining the target retrieval image provided in the present application is provided.

[0028] Figure 9 The image retrieval process comparison schematic diagram provided in the present application is provided.

[0029] Figure 10 An image retrieval flowchart provided for the present application;

[0030] Figure 11 A key target matching scenario diagram provided for the present application;

[0031] Figure 12 A visual feature encoding diagram provided for the present application;

[0032] Figure 13 A text feature encoding diagram provided for the present application;

[0033] Figure 14 An image retrieval model structure diagram provided for the present application;

[0034] Figure 15 A binary classification detection model structure diagram provided for the present application;

[0035] Figure 16 An image retrieval device structure diagram provided for the present application;

[0036] Figure 17 An electronic device structure diagram provided for the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose and implementation of the present application more clear, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all the embodiments.

[0038] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0039] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise specified. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0040] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all the components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0041] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software codes that can perform the functionality of the element.

[0042] Finally, it should be noted that the above embodiments are merely used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0043] For the convenience of explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

[0044] Figure 1 The image retrieval process provided by the present application provides a schematic diagram, including the following steps:

[0045] S101: obtaining target text information of a to-be-retrieved image, inputting the target text information into an image retrieval model, and determining a target retrieval image; wherein the image retrieval model is used to determine target text feature information of the target text information; obtaining image feature information of a plurality of images in an image library, respectively splicing the target text feature information and the image feature information of the plurality of images to obtain a plurality of image-text group feature information; and respectively determining retrieval category information corresponding to the plurality of image-text group feature information.

[0046] S102: selecting image-text group feature information corresponding to retrieval category information identical to target category information carried in the target text information, and determining a target retrieval image according to an image corresponding to the image-text group feature information.

[0047] The image retrieval method provided by the present application is applied to an electronic device, which can be a PC, a tablet computer, or the like, or a server.

[0048] The user can input target text information of an image to be retrieved to the electronic device. In a traffic violation detection scenario, the target text information includes violation category information, and can also include other information, such as vehicle attribute information, road name information, and the like. The violation category information is, for example, illegal parking, running a red light, failing to give way to pedestrians, illegal lane changing, unlicensed vehicle, illegal vehicle entering a highway, occupying an emergency lane, and the like. The vehicle attribute information is, for example, vehicle color information, vehicle model information, and the like. For example, the target text information is to detect a gray sedan running a red light.

[0049] The electronic device stores an image retrieval model trained to completion, which is used to determine retrieval category information according to input target text information. Specifically, the image retrieval model first extracts features of the target text information to determine target text feature information of the target text information. Then, image feature information of multiple images in an image library is obtained, and the target text feature information and the image feature information of the multiple images are spliced respectively to obtain multiple image-text group feature information. For example, there are 1000 pieces of image feature information of images in the image library, and the target text feature information and the image feature information of the multiple images are spliced respectively to obtain 1000 pieces of image-text group feature information. The image retrieval model extracts features of the multiple image-text group feature information to determine retrieval category information corresponding to the multiple image-text group feature information respectively. Taking a traffic violation scenario as an example, the retrieval category information is also violation type information.

[0050] After determining the retrieval category information corresponding to the multiple image-text group feature information, the image-text group feature information corresponding to the retrieval category information that is the same as the target category information carried in the target text information is selected. The target retrieval image is determined according to the image corresponding to the image-text group feature information.

[0051] In this application, the target text feature information of the target text information is determined based on the image retrieval model, and multiple image-text group feature information is spliced according to the target text feature information and image feature information of multiple images in an image library. The image retrieval model determines retrieval category information corresponding to the multiple image-text group feature information respectively, and determines the target retrieval image according to the image corresponding to the retrieval category information that is the same as the target category information carried in the target text information. Compared with the manual review and retrieval scheme, the human and material resources are greatly reduced, the image retrieval efficiency is improved, and the problem of poor accuracy caused by manual subjective retrieval that is prone to false detection or missed detection is avoided, and the accuracy of image retrieval is improved. The technical scheme provided in this application has strong robustness, reliability, operability and interpretability, and meets the trustworthy characteristics.

[0052] In the present application, the image retrieval model comprises a text feature extraction network, a visual feature extraction network and a classification network. Figure 2 The process for determining the retrieval category information corresponding to the plurality of image-text group feature information provided in the present application comprises the following steps:

[0053] S201: determining the target text feature information of the target text information based on the text feature extraction network in the image retrieval model.

[0054] S202: obtaining image feature information of a plurality of images in an image library, wherein the image feature information of the plurality of images is obtained by performing feature extraction on the plurality of images based on the visual feature extraction network in the image retrieval model, and the target text feature information and the image feature information of the plurality of images are spliced respectively to obtain a plurality of image-text group feature information.

[0055] S203: determining the retrieval category information corresponding to the plurality of image-text group feature information based on the classification network in the image retrieval model.

[0056] After the electronic device obtains the target text information, the target text information is input into the text feature extraction network in the image retrieval model, and the target text feature information of the target text information is determined based on the text feature extraction network. The image retrieval model further comprises a visual feature extraction network. After a plurality of images in an image library are input into the visual feature extraction network respectively, image feature information of the plurality of images can be obtained and saved, and the electronic device can directly obtain the image feature information of the plurality of images in the image library. Then, the target text feature information and the image feature information of the plurality of images are spliced respectively to obtain a plurality of image-text group feature information.

[0057] The image retrieval model comprises a classification network, and the plurality of image-text group feature information is input into the classification network. For the plurality of image-text group feature information, the probability value corresponding to each violation category of the image-text group feature information is determined based on the classification network in the image retrieval model, and the violation category with the maximum probability value is selected as the retrieval category information corresponding to the image-text group feature information. Optionally, if the maximum probability value is greater than a preset probability threshold, the violation category with the maximum probability value is selected as the retrieval category information corresponding to the image-text group feature information; if the maximum probability value is not greater than the preset probability threshold, it is determined that the retrieval category information corresponding to the image-text group feature information is empty. The preset probability threshold is, for example, 60%, 70%, etc.

[0058] In order to improve the efficiency of image retrieval and reduce the processing data amount of image retrieval, in the present application, obtaining the image feature information of the plurality of images in the image library comprises:

[0059] According to the predetermined correspondence between the category information and the object, a target object corresponding to target category information carried in the target text information is determined.

[0060] For the plurality of images in the image library, the objects in the images are detected, and the images not containing the target objects are filtered out to obtain image feature information of the plurality of images retained in the image library.

[0061] The predetermined correspondence between the category information and the object is, for example, the object corresponding to the category information of illegal parking is a vehicle, an illegal parking sign, a parking line, a road curb solid yellow line, and a ground net line; the object corresponding to the category information of running a red light is a traffic light and a vehicle; the object corresponding to the category information of not giving way to pedestrians is a vehicle, a pedestrian, and a sidewalk; the object corresponding to the category information of illegal lane changing is a yellow and white solid line and a vehicle; the object corresponding to the category information of a vehicle without a license plate is a vehicle and a license plate; the object corresponding to the category information of a vehicle entering a highway in violation of regulations is a vehicle; and the object corresponding to the category information of occupying an emergency lane is a vehicle and an emergency lane.

[0062] For the plurality of images in the image library, the objects in the images are detected by a target detection algorithm, and the images not containing the target objects are filtered out to obtain image feature information of the plurality of images retained in the image library. If the target object corresponding to the target category information carried in the target text information is a plurality of objects, the images not containing all the target objects are filtered out.

[0063] The present application filters the images in the image library according to the target object corresponding to the target category information carried in the target text information and the objects in the images in the image library, and only obtains image feature information of images containing all the target objects, thereby improving the efficiency of image retrieval and reducing the amount of data processed in image retrieval.

[0064] Figure 3 A process diagram for obtaining image feature information of a plurality of images in an image library provided by the present application includes the following steps:

[0065] S301: According to the predetermined correspondence between the category information and the object, a target object corresponding to target category information carried in the target text information is determined.

[0066] S302: For the plurality of images in the image library, the objects in the images are detected, and the images not containing the target objects are filtered out to obtain image feature information of the plurality of images retained in the image library.

[0067] In order to further improve the accuracy of image retrieval, in the present application, the target text feature information and the image feature information of the plurality of images are spliced to obtain a plurality of image-text group feature information, including:

[0068] The object sub-text features in the target text feature information and the object sub-image features in the image feature information of the image are input into a feature bidirectional matching model for the plurality of images, and a matching degree of the target text feature information and the image feature information of the image is determined based on the feature bidirectional matching model; if the matching degree is not greater than a preset first matching threshold, the image is filtered out; if the matching degree is greater than the preset first matching threshold, the target text feature information and the image feature information of the image are spliced to obtain image-text group feature information.

[0069] The target text feature information includes features of the original target text and object sub-text features of each object in the target text. The image feature information includes features of the original image and object sub-image features of each object in the original image. The electronic device stores a feature bidirectional matching model, which is used to determine a matching degree of the target text feature information and the image feature information. For a plurality of images, the object sub-text features in the target text feature information and the object sub-image features in the image feature information of the image are input into the feature bidirectional matching model, and the matching degree of the target text feature information and the image feature information of the image is determined based on the feature bidirectional matching model. If the matching degree is not greater than a preset first matching threshold, the image is filtered out; if the matching degree is greater than the preset first matching threshold, the target text feature information and the image feature information of the image are spliced to obtain image-text group feature information. The preset first matching threshold is, for example, 70%, 80%, etc.

[0070] Figure 4 A process diagram for determining a plurality of image-text group feature information is provided for the present application, including the following steps:

[0071] S401: The object sub-text features in the target text feature information and the object sub-image features in the image feature information of the image are input into a feature bidirectional matching model for the plurality of images, and a matching degree of the target text feature information and the image feature information of the image is determined based on the feature bidirectional matching model.

[0072] S402: If the matching degree is not greater than a preset first matching threshold, the image is filtered out; if the matching degree is greater than the preset first matching threshold, the target text feature information and the image feature information of the image are spliced to obtain image-text group feature information.

[0073] Figure 5 A training process diagram of the feature bidirectional matching model is provided for the present application, including the following steps:

[0074] S501: For a plurality of matching image-text groups in the training set, determine image feature information of the image and text feature information of the text in the matching image-text group based on the image retrieval model; wherein the text is a description text of the image, the image feature information contains object sub-image features, and the text feature information contains object sub-text features.

[0075] S502: input the object sub-text features and the object sub-image features in the matching image-text group into the feature bidirectional matching model, determine the similarity of the object sub-text features and different object sub-image features based on the feature bidirectional matching model, and determine the similarity of the object sub-image features and different object sub-text features based on the feature bidirectional matching model; according to the obtained similarities, determine the matching loss value of the feature bidirectional matching model, and train the feature bidirectional matching model according to the matching loss value.

[0076] The matching image-text group in the training set includes an image and a text describing the image. Figure 6 The matching image-text group provided in the present application is shown in the figure, which includes an image and a text Txt, as shown in Figure 6 The image has a gray vehicle parked on the sidewalk with a violation of parking sign, so the description text corresponding to the image is "Txt: a gray vehicle parked on the sidewalk with a violation of parking sign". Based on the image retrieval model, the image feature information of the image and the text feature information of the text in the matching image-text group are determined. The image feature information contains the overall image feature and the object sub-image feature, and Figure 6 For example, the object sub-image features include gray vehicle sub-image features, violation of parking sign sub-image features and sidewalk sub-image features. The text feature information contains the overall text feature and the object sub-text feature, and Figure 6 For example, the object sub-text features include "gray vehicle" sub-text features, "violation of parking sign" sub-text features and "sidewalk" sub-text features.

[0077] The object sub-text features and the object sub-image features in the matching image-text group are input into the feature bidirectional matching model, and if there are multiple object sub-text features and object sub-image features, they are all input into the feature bidirectional matching model. Based on the feature bidirectional matching model, the similarity of each object sub-text feature and different object sub-image features is determined, and the similarity of each object sub-image feature and different object sub-text features is determined.

[0078] For example, Figure 6For example, the gray vehicle sub-image feature V1, the illegal parking sign sub-image feature V2, and the pedestrian road sub-image feature V3. The "gray vehicle" sub-text feature t1, the "illegal parking sign" sub-text feature t2, and the "pedestrian road" sub-text feature t3. The similarities of V1 and t1, t2, t3, V2 and t1, t2, t3, and V3 and t1, t2, t3 are determined respectively, and the similarities of t1 and V1, V2, V3, t2 and V1, V2, V3, and t3 and V1, V2, V3 are determined respectively. For V1, t1 is a positive sample, and t2 and t3 are negative samples; for V2, t2 is a positive sample, and t1 and t3 are negative samples; for V3, t3 is a positive sample, and t1 and t2 are negative samples. For t1, V1 is a positive sample, and V2 and V3 are negative samples; for t2, V2 is a positive sample, and V1 and V3 are negative samples; for t3, V3 is a positive sample, and V1 and V2 are negative samples. According to the obtained similarities, the matching loss value of the feature bidirectional matching model is determined, and the feature bidirectional matching model is trained according to the matching loss value. The model training target is that the similarity of the positive sample is high and the similarity of the negative sample is low.

[0079] Figure 7 The training process of the image retrieval model provided in the present application is shown in the schematic diagram, which includes the following steps:

[0080] S701: For a plurality of matching image-text groups in the training set, the image feature information of the image in the matching image-text group and the text feature information of the text are determined based on the image retrieval model; the image feature information and the text feature information are spliced to obtain image-text group feature information.

[0081] S702: Determine the predicted category information corresponding to the image-text group feature information, and determine the loss value of the image retrieval model according to the predicted category information and the true category information of the matching image-text group, and train the image retrieval model according to the loss value.

[0082] The image feature information of the image in the matching image-text group is determined based on the visual feature extraction network in the image retrieval model, and the text feature information of the text in the matching image-text group is determined based on the text feature extraction network in the image retrieval model. The image feature information and the text feature information are spliced to obtain image-text group feature information. The predicted category information corresponding to the image-text group feature information is determined based on the classification network in the image retrieval model, and the loss value of the image retrieval model is determined according to the predicted category information and the true category information of the matching image-text group, and the image retrieval model is trained according to the loss value. That is, the parameters of the text feature extraction network, the visual feature extraction network, and the classification network in the image retrieval model are trained.

[0083] In the present application, in order to further improve the accuracy of image retrieval, Figure 8 The process provided in the present application for determining the target retrieval image includes the following steps:

[0084] S801: Select the image-text group feature information corresponding to the same retrieval category information as the target category information carried in the target text information as the candidate image-text group feature information.

[0085] S802: Input the candidate image-text group feature information into the binary classification detection model, determine whether the retrieval category information corresponding to the candidate image-text group feature information is the target category information based on the binary classification detection model, if yes, determine the target retrieval image according to the image corresponding to the candidate image-text group feature information, if not, filter out the candidate image-text group feature information.

[0086] In the present application, first, the image-text group feature information corresponding to the same retrieval category information as the target category information carried in the target text information is selected as the candidate image-text group feature information. The electronic device stores a trained binary classification detection model, which is used to determine whether the retrieval category information corresponding to the candidate image-text group feature information is the target category information. The binary classification detection model can be trained according to sample image-text group feature information and corresponding binary classification labels. Input the candidate image-text group feature information into the binary classification detection model, determine whether the retrieval category information corresponding to the candidate image-text group feature information is the target category information based on the binary classification detection model, if yes, determine the target retrieval image according to the image corresponding to the candidate image-text group feature information, if not, filter out the candidate image-text group feature information. According to the image corresponding to the candidate image-text group feature information, the target retrieval image is determined. Optionally, the image corresponding to the candidate image-text group feature information can be directly used as the target retrieval image.

[0087] In the present application, inputting the candidate image-text group feature information into the binary classification detection model includes:

[0088] Determine the image corresponding to the candidate image-text group feature information, splice the high-dimensional image feature information of the image and the high-dimensional target text feature information of the target text information, and input the spliced feature information into the binary classification detection model.

[0089] Optionally, the high-dimensional image feature information can be image feature information output by a second layer network layer from deep to shallow in the visual feature extraction network, and the high-dimensional target text feature information can be text feature information output by a second layer network layer from deep to shallow in the text feature extraction network. Alternatively, the high-dimensional image feature information can be image feature information output by a third layer network layer from deep to shallow in the visual feature extraction network, and the high-dimensional target text feature information can be text feature information output by a third layer network layer from deep to shallow in the text feature extraction network.

[0090] The high-dimensional image feature information of the image and the high-dimensional target text feature information of the target text information are spliced, and the spliced feature information is used as candidate image-text group feature information, so that the candidate image-text group feature information has more feature information, and the accuracy of image retrieval can be further improved.

[0091] The application provides a fast cross-membrane state retrieval method in a traffic scene. First, a target detection model is used to locate and identify target objects in a snapped traffic picture. The location and identification of the scene target help to reduce the difficulty of the retrieval task, and each necessary object in the violation scene can cause a violation event. Second, the identified target text is correlated with the necessary target text of the violation scene to obtain the attention of different objects in the image, and then the visual information feature is obtained. A text encoder is used to encode the text information, and the text feature and the visual feature are jointly trained to obtain a violation scene recognition network. Finally, the parameters of the scene recognition network are fixed, and a two-classification network of each event is trained using a violation traffic event picture, so that an accurate retrieval result of the violation event is obtained.

[0092] The application has the advantages that without forcibly performing binary coding on image features, the speed and accuracy of retrieval are improved, the target of the scene is refined, and then the refined target is mapped to a global feature space together with the text feature, so that the matching difficulty of the image-text feature is reduced. Then, high-dimensional features are used for single two-classification fine retrieval, so that more accurate retrieval results are obtained. The refined image-text object reduces the difficulty of retrieval. On the premise of maintaining the accuracy, the retrieval effect is improved. Intelligent auditing saves manpower and material resources.

[0093] Reliable technology:

[0094] Based on the image-text bidirectional matching strategy, the violation image data can be located based on the retrieval requirements of the user, and the retrieval result is more reliable and has strong robustness.

[0095] The application can locate target data in a large amount of data, and the retrieval effect is tested in multiple public data sets, and the application has reliability.

[0096] The user only needs to input the type of violation to be retrieved and add more detailed descriptions in the violation type, and specific information of vehicles, people, and scenes. The method can also be accurately positioned, is highly operable, and has interpretability.

[0097] Figure 9 A comparison diagram of the image retrieval process provided by the present application is shown in FIG. 1. Figure 9 The current process is camera snapshot, a large number of snapshot data, manual review, and giving a fine. The process of the present application is camera snapshot, a large number of snapshot data, inputting a violation event, for example, not giving way to pedestrians, violation scene retrieval, violation event discrimination, manual verification, and giving a fine.

[0098] Figure 10 A flowchart of the image retrieval process provided by the present application is shown in FIG. 2. Figure 10 The present application proposes a method for cross-membrane image retrieval in a traffic scene, which is mainly applicable to the retrieval of massive violation pictures and the positioning of violation information in a traffic scene. The present application can quickly complete the retrieval of massive data in a traffic scene without binaryzation processing of image features, and can save a large amount of auditing manpower and improve the rapid auditing and verification of violation data. Figure 10 Image embedding in FIG. 1 is a visual feature extraction network, text embedding is a text feature extraction network, MLP is a classification network, FFN is a binary classification detection model, the number of FFN is the same as the number of categories, and the bidirectional matching strategy includes a feature bidirectional matching model. Vimg0 represents the feature of the original image, Vimg1, Vimg2, and Vimg3 represent the object sub-image features of three objects in the image. t0 represents the feature of the original text, and t1, t2, and t3 represent the object sub-text features of three objects in the target text information.

[0099] First, the necessary target objects of each scene are positioned according to different traffic violation scenes. A detection model is used to position and identify the target objects (vehicles, pedestrians, sidewalks, etc.) in the snapshot traffic pictures. The positioning and identification of scene objects help to reduce the difficulty of the retrieval task, and the necessary objects exist for each violation scene to occur a violation event. Second, the relevance between the identified objects and the text information is calculated to obtain the attention of different objects in the image, and then the visual information features are obtained. A text encoder is used to encode the text information, and the text features and the visual features are jointly trained to obtain a violation scene recognition network, and pictures that do not conform to the violation scene to be retrieved are excluded. Finally, the parameters of the scene recognition network are fixed, and a binary classification network of each event is trained using the violation traffic event pictures, and the accurate retrieval result of the violation event is obtained.

[0100] The application improves an efficient illegal traffic event retrieval and auditing method in a traffic scene, which saves a lot of manpower and material resources. Unlike the binary efficient retrieval method, part of the text features is not discarded, and the accuracy and efficiency of the retrieval are improved while ensuring speed.

[0101] The data set involved in the application includes:

[0102] 1. Data set UA-DETRAC, more than 140,000 frames and 8,250 vehicles, a total of 1.21 million manually labeled boxes (precisely selected 8,350), 10 hours of video shot in 24 different locations in Beijing and Tianjin, China, recorded at a speed of 25 frames per second, with a resolution of 960x540 pixels.

[0103] 2. Data set Stanford Cars, a total of 16,185 images, 8,144 training images and 8,041 test images (precisely selected 5,100), roughly a 50-50 split for each class. The level is usually divided by manufacturer, model, and year, such as a 2012 Tesla Model S or a 2012 BMW M3 coupe.

[0104] 3. Data set CompCars, a total of 136,726 complete vehicle images, 27,618 vehicle component images, and 50,000 vehicle front views (precisely selected 5,240), complete car images are labeled with a bounding box and a viewpoint. Each vehicle model has five attributes, including maximum speed, displacement, number of doors, number of seats, and vehicle model.

[0105] 4. Data set N-CARS, 12,336 car samples, 11,693 non-car samples; 7,940 car and 7,482 background training samples, 4,396 car and 4,211 background test samples (precisely selected 5,540), a large event-based real-world data set for vehicle classification, recorded by ATIS cameras installed behind the windshield, extracted from different driving processes. Each example lasts 100 milliseconds.

[0106] Partially public traffic data sets are used to extract different traffic scene pictures, and detailed text annotations are made on the pictures. Self-built traffic violation data: 1. illegal parking 2. running a red light 3. not giving way to pedestrians 4. illegal lane changing 5. unlicensed vehicles 6. illegal vehicles entering the highway 7. occupying the emergency lane 8. 9. Each violation scene constructs a 4,000 violation data set, and a 2,000 scene similar but not violating data set.

[0107] The main body of the network of the image retrieval model adopts an encoder in a Transformer structure to encode visual picture information and text information. It is currently verified that the Transformer is superior to the conventional convolutional neural network in the visual field and natural language processing. In order to support various retrieval requirements of the traffic scene, not only the features of the entire violation scene need to be concerned, but also the key targets in the scene need to be concerned, which are consistent with the key targets that the user wants to retrieve, which requires the consistency alignment of the visual targets and the text targets.

[0108] Image target and text target consistency check: a lightweight detection model such as the yolo series is adopted, and the inference speed of the YOLOv8n model on A100 through TensorRT acceleration reaches 1ms per frame of processing speed, and the processing performance is very good. The occurrence of each scene event must contain certain necessary factor targets, for example, the pedestrian-unfriendly sidewalk event must contain three target objects, pedestrians, sidewalks, and vehicles, and the event does not exist if one factor is missing. Therefore, by using this visual target and text target consistency determination method, a large amount of non-scene data can be eliminated, the scene data is refined, and the pressure of model retrieval is effectively reduced, and the retrieval efficiency is improved.

[0109] Construct a key target library: {vehicle, illegal parking sign, parking line, road curb yellow solid line, ground network line, pedestrian, sidewalk, yellow and white single solid line, double yellow line, unlicensed vehicle, emergency lane, traffic light}, according to the increase of violation events, new key targets can be added to supplement the key elements of the target library.

[0110] Figure 11 The key target matching scene diagram provided for the present application retains the images for which the key target matching is successful, and directly deletes the images for which the key target matching is unsuccessful.

[0111] The visual feature extraction is as follows:

[0112] In the visual scene, there are often many targets, and it is difficult to achieve good retrieval effect when matching with text features due to the heterogeneous nature of the two types of information. In order to reduce the difficulty of retrieval, the targets in the vision are refined and input into the feature extraction network as a visual target. This method significantly reduces the difficulty of retrieval and extracts global visual features and target visual features together. Whether an event occurs mainly depends on the spatial relationship of the target feature level target feature.

[0113] The input of the standard transformer is a one-dimensional word vector. In order to process a two-dimensional image, the image is divided into m*n small patches, each of which is p*p in size. The number of patches affects the length of the input sequence. Each patch is flattened into a fixed-length vector such as 1024 using linear mapping, and position encoding is added. Each patch is input into the multi-head attention mechanism to encode the visual image. By linear mapping to the same dimensional space, the final visual feature vectors Vimg0, Vimg1, Vimg2, and Vimg3 are obtained, where Vimg0 is the feature vector of the original image, and Vimg1, Vimg2, and Vimg3 are the object sub-visual vectors. Figure 12 A visual feature encoding diagram is provided for the present application.

[0114] Figure 13 A text feature encoding diagram is provided for the present application. The text is key-subdivided to obtain multiple text fields. Each field is encoded by a text encoder to obtain multiple text feature vectors, which are linearly mapped to the same dimensional space. Where t0 is the original text feature vector, and t1, t2, and t3 are the object sub-text feature vectors.

[0115] Text-visual feature matching loss:

[0116] The effect of feature bidirectional matching: When searching for an image and text, the objects described by the text and the objects in the image need to be one-to-one corresponding in order to be considered as the target to be searched. In one case, there are many visual targets in the image, and it is difficult to get good search results for the matching of the entire image and text features. Therefore, we refine the target and find the target described by the text in the visual target, as well as the target appearing in the text in the visual target. In this way, the search of the entire image and text is converted into the matching search of the target in the visual image and the target in the text. Therefore, the target in the image is extracted and then searched in detail with the target extracted from the text, effectively reducing the search difficulty of the entire image and the entire text, and effectively improving the search accuracy. This paper proposes a bidirectional matching strategy to match the target in the text and the visual target. The main purpose is to align two heterogeneous features, which is beneficial to improve the accuracy of the back-end classification network, reduce the difficulty of classification, and achieve good search results.

[0117] Matching loss: If (V1, t1) is the corresponding matching visual-text pair, then for v1, t1 is the positive target sample, and t2, t3, and other text targets are negative target samples. Sv2t(v) is the similarity calculation function of the image target and the text, and St2v(t) is the similarity calculation function of the text target and the image target. R is the one-hot encoding of the true label. Then the image target and the text target are matched by calculating the cross-entropy between R and S. After full connection, it is assumed that v is an n-dimensional vector, and t is also an n-dimensional vector:

[0118]

[0119] where the v1 vector is (x1, x2, …, xn), and the ti vector is (yi1, yi2, …, yin).

[0120]

[0121] H is the cross-entropy function. The loss is continuously optimized to match the visual target and the text target. According to the matching score, different proportions are set, the visual target features are fused, the text features are fused, and then input into the joint training network for classification training.

[0122] According to the matching result, multiple video features and multiple text features are sequentially spliced according to the matching corresponding relationship, and the visual target is spliced into a vector, and the text target is spliced into a vector, which are input into the feature mapping and classification network, effectively improving the retrieval efficiency.

[0123] Classification loss:

[0124] Multiple visual targets are spliced into a vector, multiple text targets are spliced into a vector, and the two features are spliced for joint training of the initial classification network. Through full connection, the visual and text feature vectors are mapped to a global space for initial classification based on violation scenes, and further irrelevant data is removed.

[0125] Classification by MLP again:

[0126] Input layer: The number of neurons in the input layer is equal to the number of features of the input data, and no calculation is performed. Assume that the input data is x.

[0127] Hidden layer: Assuming that the hidden layer has n layers and the output is F(x), the output calculation can be represented as:

[0128] F(x) = f(W(n) * f(W(n-1) * … * f(W(1) * x + b(1)) … + b(n-1)) + b(n));

[0129] x is the vector of visual text splicing, W(n) is the weight, and b is the bias.

[0130] where W(n) is the weight matrix of the nth layer, b(n) is the bias vector of the nth layer, and f is the activation function. Common activation functions include sigmoid, ReLU, tanh, etc.

[0131] Output layer: assuming the input of the output layer is F(x) and the output is Ycls, the calculation of the output layer can be represented as: Ycls = softmax(F(x));

[0132] where W(L) is the weight matrix of the output layer, b(L) is the bias vector of the output layer, and softmax is a normalization function that converts the output into a probability distribution.

[0133] FFN layer binary classification: fix the high-dimensional feature parameters of the classification network, use the violation data to train the FFN layer for binary classification, and perform fine screening on the data. Commonly used in binary classification tasks, calculate the cross-entropy of the predicted probability distribution and the true label. Binary classification loss function:

[0134] Loss_2cls = -[C*log(P) + (1-C)*log(1-P)];

[0135] where C is the true label and P is the probability value of the predicted label.

[0136] Figure 14 The image retrieval model structure diagram provided in the present application, Image embedding is a visual feature extraction network, Text embedding is a text feature extraction network, and MLP is a classification network. Scene classification network training: first, use 24,230 regular traffic scene data in the public dataset and part of the violation data to train the initial classification network 8,000 times, and the parameter settings for training are bachsize = 32, epoch = 150, learning_rate = 0.0003, optimizer = Adam, num_layer = 6, d_model = 1024, num_heads = 8, and dropout = 0.1.

[0137] Figure 15The binary classification detection model structure diagram provided by the application is used to train the binary classification network, extract the high-dimensional features of the full connection layer, and fix the network parameters before the layer. Only 3000 pieces of violation data and 2000 pieces of non-violation data of each category are used, and 2000 pieces of similar scene data are added for secondary training. The main purpose is to improve the accurate identification performance of the binary classification network, further reduce the category range, and clearly define the category area boundary. Bachsize = 16, epoch = 100, and learning_rate = 0.001.

[0138] Figure 16 The image retrieval device structure diagram provided by the application includes:

[0139] The first determination module 161 is configured to obtain target text information of a to-be-retrieved image, input the target text information into an image retrieval model, and determine a target retrieval image. The image retrieval model is configured to determine target text feature information of the target text information. Image feature information of a plurality of images in an image library is obtained. The target text feature information and the image feature information of the plurality of images are spliced respectively to obtain a plurality of image-text group feature information. The retrieval category information corresponding to the plurality of image-text group feature information is determined respectively.

[0140] The second determination module 162 is configured to select image-text group feature information corresponding to retrieval category information identical to target category information carried in the target text information, and determine a target retrieval image according to an image corresponding to the image-text group feature information.

[0141] The first determination module 161 is configured to determine target text feature information of the target text information based on a text feature extraction network in the image retrieval model. Image feature information of a plurality of images in an image library is obtained. The image feature information of the plurality of images is obtained by performing feature extraction on the plurality of images based on a visual feature extraction network in the image retrieval model. The target text feature information and the image feature information of the plurality of images are spliced respectively to obtain a plurality of image-text group feature information. The retrieval category information corresponding to the plurality of image-text group feature information is determined respectively based on a classification network in the image retrieval model.

[0142] The first determination module 161 is configured to determine a target object corresponding to target category information carried in the target text information according to a predetermined correspondence between category information and objects. For a plurality of images in the image library, objects in the images are detected, images not containing the target object are filtered out, and image feature information of a plurality of images retained in the image library is obtained.

[0143] The first determination module 161 is configured to, for the plurality of images, input object sub-text features in the target text feature information and object sub-image features in image feature information of the images into a feature bidirectional matching model, determine a matching degree of the target text feature information and the image feature information of the images based on the feature bidirectional matching model, filter out the images if the matching degree is not greater than a preset first matching threshold, and splice the target text feature information and the image feature information of the images if the matching degree is greater than the preset first matching threshold, to obtain image-text group feature information.

[0144] The first determination module 161 is configured to, for a plurality of matching image-text groups in a training set, determine image feature information of images and text feature information of texts in the matching image-text groups based on the image retrieval model, wherein the texts are description texts of the images, the image feature information includes object sub-image features, and the text feature information includes object sub-text features; input the object sub-text features and the object sub-image features in the matching image-text groups into the feature bidirectional matching model, respectively determine similarities of the object sub-text features and different object sub-image features based on the feature bidirectional matching model, and respectively determine similarities of the object sub-image features and different object sub-text features; determine a matching loss value of the feature bidirectional matching model according to the obtained similarities, and train the feature bidirectional matching model according to the matching loss value.

[0145] The first determination module 161 is configured to, for a plurality of matching image-text groups in a training set, determine image feature information of images and text feature information of texts in the matching image-text groups based on the image retrieval model, splice the image feature information and the text feature information to obtain image-text group feature information, determine predicted category information corresponding to the image-text group feature information, determine a loss value of the image retrieval model according to the predicted category information and real category information of the matching image-text groups, and train the image retrieval model according to the loss value.

[0146] The second determination module 162 is configured to select image-text group feature information corresponding to retrieval category information same as target category information carried in the target text information as candidate image-text group feature information, input the candidate image-text group feature information into a binary classification detection model, determine whether retrieval category information corresponding to the candidate image-text group feature information is the target category information based on the binary classification detection model, determine a target retrieval image according to an image corresponding to the candidate image-text group feature information if yes, and filter out the candidate image-text group feature information if no.

[0147] The second determination module 162 is configured to determine an image corresponding to the candidate image-text group feature information, splice high-dimensional image feature information of the image and high-dimensional target text feature information of the target text information, take the spliced feature information as candidate image-text group feature information, and input the candidate image-text group feature information into the binary classification detection model.

[0148] The present application also provides an electronic device, such as Figure 17 As shown in the figure, the electronic device comprises a processor 171, a communication interface 172, a memory 173 and a communication bus 174, wherein the processor 171, the communication interface 172 and the memory 173 are in communication with each other through the communication bus 174.

[0149] The memory 173 stores a computer program, and when the program is executed by the processor 171, the processor 171 executes any of the above method steps.

[0150] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0151] The communication interface 172 is used for communication between the above-mentioned electronic device and other devices.

[0152] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0153] The processor mentioned above can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0154] The application also provides a computer storage readable storage medium, which stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device is caused to execute the above any method steps.

[0155] Although preferred embodiments of the application have been described herein, additional changes and modifications can be made by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such changes and modifications that fall within the scope of the application.

[0156] Obviously, various modifications and changes can be made to the application without departing from the spirit and scope thereof. Accordingly, it is intended that the application embrace all such modifications and changes as fall within the scope of the appended claims and their equivalents.

Claims

1. An image retrieval method characterized by, The method comprises: acquiring target text information of an image to be retrieved, inputting the target text information into an image retrieval model, and determining a target retrieval image; The image retrieval model is used to determine target text feature information of the target text information; image feature information of a plurality of images in an image library is acquired, the target text feature information and the image feature information of the plurality of images are spliced respectively, and a plurality of image-text group feature information is obtained; and retrieval category information corresponding to the plurality of image-text group feature information is determined respectively. image-text group feature information corresponding to retrieval category information identical to target category information carried in the target text information is selected, and a target retrieval image is determined according to an image corresponding to the image-text group feature information; The method comprises: The image retrieval model comprises a text feature extraction network, a visual feature extraction network, and a classification network; and the method comprises:

2. The method of claim 1, wherein, target text feature information of the target text information is determined based on the text feature extraction network in the image retrieval model; image feature information of a plurality of images in an image library is acquired, wherein the image feature information of the plurality of images is obtained by performing feature extraction on the plurality of images based on the visual feature extraction network in the image retrieval model, and the target text feature information and the image feature information of the plurality of images are spliced respectively, and a plurality of image-text group feature information is obtained; retrieval category information corresponding to the plurality of image-text group feature information is determined based on the classification network in the image retrieval model. The method comprises:

3. The method of claim 1, wherein, a target object corresponding to target category information carried in the target text information is determined according to a predetermined correspondence between category information and objects; for a plurality of images in the image library, objects in the images are detected, images not containing the target object are filtered out, and image feature information of a plurality of images remaining in the image library is acquired. The training process of the feature bidirectional matching model comprises:

4. The method of claim 1, wherein, for a plurality of matching image-text groups in a training set, image feature information and text feature information of the matching image-text groups are determined based on the image retrieval model; wherein the text is a description text of the image, the image feature information contains object sub-image features, and the text feature information contains object sub-text features; ​ The object subtext features and the object subimage features in the matching image-text group are input into the feature bidirectional matching model, similarity of the object subtext features and different object subimage features is respectively determined based on the feature bidirectional matching model, and similarity of the object subimage features and different object subtext features is respectively determined; a matching loss value of the feature bidirectional matching model is determined according to the obtained similarities, and the feature bidirectional matching model is trained according to the matching loss value.

5. The method of claim 1, wherein, The training process of the image retrieval model comprises: For a plurality of matching image-text groups in a training set, image feature information of images and text feature information of texts in the matching image-text groups are determined based on the image retrieval model; the image feature information and the text feature information are spliced to obtain image-text group feature information; prediction category information corresponding to the image-text group feature information is determined, a loss value of the image retrieval model is determined according to the prediction category information and real category information of the matching image-text groups, and the image retrieval model is trained according to the loss value.

6. The method of claim 1, wherein, The image-text group feature information corresponding to the same retrieval category information as the target category information carried in the target text information is selected, and a target retrieval image is determined according to an image corresponding to the image-text group feature information, which comprises: The image-text group feature information corresponding to the same retrieval category information as the target category information carried in the target text information is selected as candidate image-text group feature information. The candidate image-text group feature information is input into a binary classification detection model, whether the retrieval category information corresponding to the candidate image-text group feature information is the target category information is determined based on the binary classification detection model, if yes, a target retrieval image is determined according to an image corresponding to the candidate image-text group feature information, and if no, the candidate image-text group feature information is filtered out.

7. The method of claim 6, wherein, The candidate image-text group feature information is input into a binary classification detection model, which comprises: An image corresponding to the candidate image-text group feature information is determined, high-dimensional image feature information of the image and high-dimensional target text feature information of the target text information are spliced, the spliced feature information is taken as candidate image-text group feature information, and the candidate image-text group feature information is input into a binary classification detection model.

8. An image retrieval apparatus characterized by comprising: The device comprises: A first determination module is configured to acquire target text information of a to-be-retrieved image, input the target text information into an image retrieval model, and determine a target retrieval image; the image retrieval model is configured to determine target text feature information of the target text information; acquire image feature information of a plurality of images in an image library, splice the target text feature information and the image feature information of the plurality of images respectively to obtain a plurality of image-text group feature information; and determine retrieval category information corresponding to the plurality of image-text group feature information respectively; A second determination module is configured to select image-text group feature information corresponding to the same retrieval category information as target category information carried in the target text information, and determine a target retrieval image according to an image corresponding to the image-text group feature information. The first determining module is configured to, for the plurality of images, input an object sub-text feature in the target text feature information and an object sub-image feature in image feature information of the image into a feature bidirectional matching model, determine a matching degree of the target text feature information and the image feature information based on the feature bidirectional matching model, filter out the image if the matching degree is not greater than a preset first matching threshold, and splice the target text feature information and the image feature information if the matching degree is greater than the preset first matching threshold, to obtain image-text group feature information.

9. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored on the memory to implement the method steps in any one of claims 1-7.

Images