A human-robot hybrid intelligent interactive dynamic person portrait retrieval method

Abstract

The present application belongs to the field of dynamic face retrieval, and relates to an interactive dynamic portrait retrieval method of human-computer hybrid intelligence, comprising: obtaining a sketch drawn by an artist, inputting the sketch into a generation model to obtain a generated portrait; inputting the sketch and the generated portrait into a deep neural network model respectively to obtain embedding vectors and key point information of the current sketch and the generated portrait and to fuse them; calculating matching scores of the fused embedding vectors and joint key point information with embedding vectors and key point information of all portraits to be retrieved, sorting the portraits to be retrieved in ascending order according to the matching scores, and displaying top-k portraits in real time to the artist; the present application adopts a generation joint retrieval framework, infers and generates a complete image representation from limited sketch images, can fill in missing details in the sketch, enhances the information amount of the original sketch, provides more visual features available for comparison and matching for retrieval, and thus improves the accuracy and efficiency of retrieval.

Description

Technical Field

[0001] This invention belongs to the field of dynamic face retrieval and relates to an interactive dynamic portrait retrieval method with human-computer hybrid intelligence. Background Technology

[0002] The rapid growth of data has made retrieving target information increasingly difficult, leading to the development of various retrieval technologies. Current retrieval technologies mainly include text-based retrieval, content-based retrieval, and retrieval techniques based on hand-drawn portraits.

[0003] Traditional retrieval techniques based on hand-drawn portraits typically involve: (1) the artist listens to eyewitness descriptions or watches low-quality video footage; (2) the artist draws a complete portrait sketch of the target person; (3) the eyewitness searches for and matches the target person based on the artist's hand-drawn portrait image; and (4) if the match fails, the portrait image is revised and redrawn until the target person is identified.

[0004] Patent "2023102295139" utilizes a real-time retrieval method for hand-drawn images based on multimodal data fusion. The method performs real-time retrieval based on the number of strokes drawn by the artist each time, obtains retrieval results, and the artist draws the next few strokes based on the retrieval results until the target character appears. The method provides a human-computer hybrid interaction approach that improves retrieval efficiency.

[0005] However, in the aforementioned existing technologies, the sketch itself only contains black and white colors, has simple lines, contains little information, and lacks sufficient sketch information in the early stages of retrieval, resulting in low retrieval accuracy. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an interactive dynamic portrait retrieval method based on human-machine hybrid intelligence, characterized by comprising: acquiring a sketch drawn by an artist; inputting the sketch drawn by the artist into a trained generative joint retrieval model for real-time retrieval to obtain retrieval results; wherein the generative joint retrieval model comprises: a generative model and a deep neural network model;

[0007] The training process for generating the joint retrieval model includes:

[0008] S1. Construct a portrait-sketch dataset and build mini-batch training data based on the portrait-sketch dataset. The small batch training data Including sketch sequences Sketch sequence Corresponding positive sample portrait sequence and negative sample portrait sequences ;

[0009] S2, Sketch sequence Input the generative model to get the generated portrait. ;

[0010] S3, Sketch sequence Generate portrait Positive sample portrait sequence and negative sample portrait sequences Inputting each into a deep neural network model yields the following results: Embedded vector and key information , Embedded vector and key information , Embedded vector and key information as well as Embedded vector and key information ;

[0011] S4, Fusion Embedded Vector and embedding vector The joint embedding vector is obtained. Integrate key information and key information obtain joint key point information ;

[0012] S5. Based on the joint embedding vector Joint key point information Embedded vectors Key information Embedded vectors and key information Calculate the loss function value, update the parameters of the joint retrieval model based on the loss function value, and complete the model training when the preset number of training iterations is reached.

[0013] Mini-batch training data was constructed based on the portrait-sketch dataset. include:

[0014] Random sampling from the portrait-sketch dataset Zhang's sketches were used as small-batch training data. Sketch sequence ,Will Zhang's sketch corresponds to The portrait photos were used as a small batch of training data. Positive sample portrait sequence In the portrait-sketch dataset, portrait sequences that are different from the positive samples are randomly selected. Each portrait in The portrait photos were used as a small batch of training data. Negative sample portrait sequences .

[0015] The deep neural network model includes: a first convolutional network layer. Attention layer First dimensionality reduction layer Second convolutional network layer and the second dimensionality reduction layer Deep neural network models for sketch sequences Generate portrait Positive sample portrait sequence and negative sample portrait sequences The processing steps include:

[0016] S31, Sketch sequence Generate portrait Positive sample portrait sequence and negative sample portrait sequences All are input into the first convolutional network layer Second convolutional network layer ;

[0017] S32, the first convolutional network layer The output of the input attention layer Attention layer The output of the input to the first dimensionality reduction layer ,get Embedded vector , Embedded vector , Embedded vector as well as Embedded vector ;

[0018] S33, Add the second convolutional network layer The output is input into the second dimensionality reduction layer. ,get Key information , Key information , Key information as well as Key information .

[0019] loss function for:

[0020]

[0021] in, Indicates retrieval loss. Indicates key point losses. , As weight.

[0022] The critical point loss is:

[0023]

[0024] in, Represents mini-batch training data Number of sketches in Chinese For sketch sequence The joint key point information of the i-th sketch in the image. Positive sample portrait sequence Key information of the i-th portrait in the image. negative sample portrait sequence Key information of the i-th portrait.

[0025] Loss of portrait retrieval for:

[0026]

[0027] in, For sketch sequence The joint embedding vector of the i-th sketch in the dataset. Positive sample portrait sequence The embedding vector of the i-th portrait in the image. negative sample portrait sequence The embedding vector of the i-th portrait in the image. Represents mini-batch training data The number of sketches in the middle.

[0028] Obtain sketches drawn by artists, and input these sketches into a trained joint retrieval model for real-time retrieval, including:

[0029] Step 1: Obtain the portrait dataset. Input the portraits from the dataset into a trained deep neural network model to obtain the embedding vector database of the portrait dataset. and key information database ;in, Let be the embedding vector of the j-th portrait. For the key information of the j-th portrait, ;

[0030] Step 2: Obtain the sketch drawn by the artist, input the sketch drawn by the artist into the joint retrieval model, and obtain the joint embedding vector and joint key point information of the sketch.

[0031] Step 3: Based on the joint embedding vector and joint keypoint information of the sketch, and the embedding vector database of the portrait dataset... and key information Calculate the matching score for each portrait in the sketch and portrait dataset;

[0032] Step 4: Sort the portraits in the portrait dataset in ascending order according to the matching score, and display the top-k portrait photos to the artist in real time;

[0033] Step 5: The artist determines whether the target person appears in the top-k portrait photos. If the target person does not appear, the artist draws the next few strokes of the sketch based on the top-k portrait photos and returns to step S2; otherwise, the final portrait of the target person is obtained.

[0034] Calculating the matching score for each portrait in the sketch and portrait dataset includes:

[0035]

[0036] in, For the first in the sketch and portrait dataset Zhang's portrait matching score Represents the joint embedding vector of the sketch. For the joint key point information of the sketch, Represents the first of the portrait datasets Embedding vector of Zhang's portrait, For the portrait dataset Key information about Zhang's portrait , As weight.

[0037] The generative model is a diffusion model.

[0038] Beneficial effects:

[0039] 1. This invention employs a generative joint retrieval framework to infer and generate complete image representations from limited sketch images. This fills in missing details in the sketches, enhances the information content of the original sketches, and provides more visual features for comparison and matching, thereby improving the accuracy and efficiency of retrieval. 2. This invention uses a human-computer hybrid interactive approach for portrait retrieval. After each retrieval result from the model, the artist can refer to the results. It does not rely on highly skilled artists; ordinary artists or even eyewitnesses can complete the portrait drawing, retrieval, and confirmation work, thus improving retrieval efficiency. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the generation of a joint retrieval model provided in an embodiment of the present invention;

[0041] Figure 2 A flowchart illustrating an interactive dynamic portrait retrieval method using human-machine hybrid intelligence, provided as an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, the present invention employs an interactive dynamic portrait retrieval method with human-machine hybrid intelligence, comprising: acquiring a sketch drawn by an artist; inputting the sketch drawn by the artist into a trained generative joint retrieval model for real-time retrieval; and obtaining retrieval results; wherein the generative joint retrieval model comprises: a generative model and a deep neural network model;

[0044] The training process for generating the joint retrieval model includes:

[0045] S1. Construct a portrait-sketch dataset and build mini-batch training data based on the portrait-sketch dataset. The small batch training data Including sketch sequences Sketch sequence Corresponding positive sample portrait sequence and negative sample portrait sequences ;

[0046] The construction of the portrait-sketching dataset includes:

[0047] The portrait-sketch dataset Set={I, S} used for training is obtained from the existing FS2K-SDE dataset, specifically consisting of complete portrait photographs. And the corresponding images rendered in chronological order as a sequence of portrait sketches. Composition, in which Indicates that there is [Number of pictures]

[0048] The data in the portrait-sketch dataset is augmented by rendering the corresponding image sketches to the required number of strokes to draw a complete sketch. Zhang's sketch sequence, obtaining each portrait sketch sequence After rendering, the images in the dataset are enhanced by rotation, scaling, cropping, etc.

[0049] Mini-batch training data was constructed based on the portrait-sketch dataset. include:

[0050] Random sampling from the portrait-sketch dataset Zhang's sketches were used as small-batch training data. Sketch sequence ,Will Zhang's sketch corresponds to The portrait photos were used as a small batch of training data. Positive sample portrait sequence In the portrait-sketch dataset, portrait sequences that are different from the positive samples are randomly selected. Each portrait photograph in the dataset is used as a mini-batch training data. negative sample portrait sequence .

[0051] S2, Sketch sequence Input the generative model to get the generated portrait. ;

[0052] S3, Sketch sequence Generate portrait Positive sample portrait sequence and negative sample portrait sequences Inputting each into a deep neural network model yields the following results: Embedded vector and key information , Embedded vector and key information , Embedded vector and key information as well as Embedded vector and key information ;

[0053] The deep neural network model includes: a first convolutional network layer. Attention layer First dimensionality reduction layer Second convolutional network layer and the second dimensionality reduction layer ;

[0054] Deep neural network models for sketch sequences Generate portrait Positive sample portrait sequence and negative sample portrait sequences The processing steps include:

[0055] S31, Sketch sequence Generate portrait Positive sample portrait sequence and negative sample portrait sequences All are input into the first convolutional network layer Second convolutional network layer ;

[0056] S32, the first convolutional network layer The output of the input attention layer Attention layer The output result is input into the first dimensionality reduction layer. ,get Embedded vector , Embedded vector , Embedded vector as well as Embedded vector ;

[0057] S33, Add the second convolutional network layer The output is input into the second dimensionality reduction layer. ,get Key information , Key information , Key information as well as Key information .

[0058] S4, Fusion Embedded Vector and embedding vector , that is, embedding vector and embedding vector The feature vectors are summed along their dimensions to obtain the joint embedding vector. Integrate key information and key information obtain joint key point information ;

[0059] Dimensionality reduction layer The specific process includes: reducing the embedded vector sequence output by the attention layer through linear dimensionality reduction and then passing it through another attention layer to complete the final dimensionality reduction.

[0060] S5. Based on the joint embedding vector Joint key point information Embedded vectors Key information Embedded vectors and key information Calculate the loss function value, update the parameters of the joint retrieval model based on the loss function value, and complete the model training when the preset number of training iterations is reached.

[0061] loss function for:

[0062]

[0063] Key point loss ( )for:

[0064]

[0065] Portrait retrieval loss ( )for:

[0066]

[0067] in, Indicates retrieval loss. Indicates key point losses. , As weight, Represents mini-batch training data Number of sketches in Chinese For sketch sequence The joint key point information of the i-th sketch in the image. Positive sample portrait sequence Key information of the i-th portrait in the image. negative sample portrait sequence Key information of the i-th portrait in the image. For sketch sequence The joint embedding vector of the i-th sketch in the dataset. Positive sample portrait sequence The embedding vector of the i-th portrait in the image. negative sample portrait sequence The embedding vector of the i-th portrait in the image. Represents mini-batch training data The number of sketches in the middle.

[0068] The generative model can be a diffusion model pre-trained based on a portrait-sketch dataset.

[0069] like Figure 2As shown, the process involves obtaining sketches drawn by artists, inputting these sketches into a trained joint retrieval model for real-time retrieval, including:

[0070] Step 1: Input the entire dataset of portraits to be retrieved into the trained convolutional network layer. Attention layer Dimensionality reduction layer In the process, the embedding vectors of all the portrait photos to be retrieved are obtained, and the entire dataset of portrait photos to be retrieved is input into the trained convolutional network layer. Dimensionality reduction layer Key point information is extracted from portrait images, and a database is constructed based on this information for easy querying and sorting; the database includes: an embedded vector database. and key information database ;in, Let be the embedding vector of the j-th portrait. For the key information of the j-th portrait, ;

[0071] Step 2: Using the portrait sketch drawn by the user on the canvas provided by the software as the original sketch, each additional stroke in the stroke order of the drawing will create a new sketch image.

[0072] Step 3: Perform data augmentation on the user-drawn portrait sketch photo, input the augmented sketch photo into the generative model for inference to generate the portrait image, and then input the portrait sketch of the current stroke and the generated portrait image into the trained convolutional network layer. Attention layer Modules, dimensionality reduction layers The embedding vectors of the current sketch image and the generated portrait image are obtained and then input into the trained convolutional network layer. Dimensionality reduction layer Obtain key point information from the current sketch image and the generated portrait image, and then fuse them;

[0073] Step 4: Calculate the matching score between the fused embedding vector and key point information of the current stroke's sketch image and the embedding vectors and key point information of all portrait photos to be retrieved.

[0074] Step 5: Sort the portraits in the portrait dataset in ascending order according to the matching score, and display the top-k portrait photos to the artist in real time;

[0075] Step 6: The artist determines whether the target person appears in the top-k portrait photos. If the target person does not appear, the artist draws the next few strokes of the sketch based on the top-k portrait photos and returns to step 3; otherwise, the final portrait of the target person is obtained.

[0076] Calculating the matching score for each portrait in the sketch and portrait dataset includes:

[0077]

[0078] in, For the first in the sketch and portrait dataset Zhang's portrait matching score Represents the joint embedding vector of the sketch. For the joint key point information of the sketch, Represents the first of the portrait datasets Embedding vector of Zhang's portrait, For the portrait dataset Key information about Zhang's portrait , As weight.

[0079] When there is no portrait photo of the target person and the text is insufficient to describe the target person's features, users can retrieve the target person by drawing a portrait on a touchscreen device based on their image of the target person. The user-drawn portrait sketch is then input into a trained deep neural network model, which retrieves k images that are most similar to the portrait sketch.

[0080] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human-machine hybrid intelligent interactive dynamic portrait retrieval method, characterized in that, include: Obtain the sketches drawn by the artist, input the sketches drawn by the artist into the trained generative joint retrieval model for real-time retrieval, and obtain the retrieval results; The generative joint retrieval model includes: a generative model and a deep neural network model; The training process for generating the joint retrieval model includes: S1. Construct a portrait-sketch dataset and build mini-batch training data based on the portrait-sketch dataset. The small batch training data Including sketch sequences Sketch sequence Corresponding positive sample portrait sequence and negative sample portrait sequences ; S2, Sketch sequence Input the generative model to get the generated portrait. ; S3, Sketch sequence Generate portrait Positive sample portrait sequence and negative sample portrait sequences Inputting each into a deep neural network model yields the following results: Embedded vector and key information , Embedded vector and key information , Embedded vector and key information as well as Embedded vector and key information ; S4, Fusion Embedded Vector and embedding vector The joint embedding vector is obtained. Integrating key information and key information obtain joint key point information ; S5. Based on the joint embedding vector Joint key point information Embedded vectors Key information Embedded vectors and key information Calculate the loss function value, update the parameters of the joint retrieval model based on the loss function value, and complete the model training when the preset number of training iterations is reached. loss function for: in, Indicates retrieval loss. Indicates key point losses. , As weight; The critical point loss is: in, Represents mini-batch training data Number of sketches in Chinese For sketch sequence The joint key point information of the i-th sketch in the image. Positive sample portrait sequence Key information of the i-th portrait in the image. negative sample portrait sequence Key information of the i-th portrait in the image; Search loss for: in, For sketch sequence The joint embedding vector of the i-th sketch in the dataset. Positive sample portrait sequence The embedding vector of the i-th portrait in the image. negative sample portrait sequence The embedding vector of the i-th portrait in the image.

2. The interactive dynamic portrait retrieval method of human-machine hybrid intelligence according to claim 1, characterized in that, Mini-batch training data was constructed based on the portrait-sketch dataset. include: Random sampling from the portrait-sketch dataset Zhang's sketches were used as small-batch training data. Sketch sequence ,Will Zhang's sketch corresponds to The portrait photos were used as a small batch of training data. Positive sample portrait sequence In the portrait-sketch dataset, portrait sequences that are different from the positive samples are randomly selected. Each portrait in The portrait photos were used as a small batch of training data. Negative sample portrait sequences .

3. The interactive dynamic portrait retrieval method of human-machine hybrid intelligence according to claim 1, characterized in that, The deep neural network model includes: a first convolutional network layer. Attention layer First dimensionality reduction layer Second convolutional network layer and the second dimensionality reduction layer ; Deep neural network models for sketch sequences Generate portrait Positive sample portrait sequence and negative sample portrait sequences The processing steps include: S31, Sketch sequence Generate portrait Positive sample portrait sequence and negative sample portrait sequences All are input into the first convolutional network layer Second convolutional network layer ; S32, the first convolutional network layer The output of the input attention layer Attention layer The output result is input into the first dimensionality reduction layer. ,get Embedded vector , Embedded vector , Embedded vector as well as Embedded vector ; S33, Add the second convolutional network layer The output is input into the second dimensionality reduction layer. ,get Key information , Key information , Key information as well as Key information .

4. The interactive dynamic portrait retrieval method of human-machine hybrid intelligence according to claim 1, characterized in that, Obtain sketches drawn by artists, and input these sketches into a trained joint retrieval model for real-time retrieval, including: Step 1: Obtain the portrait dataset. Input the portraits from the dataset into a trained deep neural network model to obtain the embedding vector database of the portrait dataset. and key information database ;in, Let be the embedding vector of the j-th portrait. For the key information of the j-th portrait, ; Step 2: Obtain the sketch drawn by the artist, input the sketch drawn by the artist to generate a joint retrieval model, and obtain the joint embedding vector and joint key point information of the sketch. Step 3: Based on the joint embedding vector and joint keypoint information of the sketch, and the embedding vector database of the portrait dataset... and key information Calculate the matching score for each portrait in the sketch and portrait dataset; Step 4: Sort the portraits in the portrait dataset in ascending order according to the matching score, and display the top-k portrait photos to the artist in real time; Step 5: The artist determines whether the target person appears in the top-k portrait photos. If the target person does not appear, the artist draws the next few strokes of the sketch based on the top-k portrait photos and returns to step 2; otherwise, the final portrait of the target person is obtained.

5. The human-machine hybrid intelligent interactive dynamic portrait retrieval method according to claim 4, characterized in that, Calculating the matching score for each portrait in the sketch and portrait dataset includes: in, For the first in the sketch and portrait dataset Zhang's portrait matching score Represents the joint embedding vector of the sketch. For the joint key point information of the sketch, Represents the first of the portrait datasets Embedding vector of Zhang's portrait, For the portrait dataset Key information about Zhang's portrait , For weights.

6. The interactive dynamic portrait retrieval method of human-machine hybrid intelligence according to claim 1, characterized in that, The generative model is a diffusion model.

Images