Image query method and device, computer equipment, storage medium and program product
Through the text recognition network and image classification network of the image query model, the image features are automatically extracted and matched, solving the problem of inefficiency of traditional image query methods and achieving fast and efficient image query.
Patent Information
- Application Number
- CN202411660026.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-03-07
AI Technical Summary
Traditional image query methods require manual image feature extraction, resulting in inefficient query.
An image query method is provided, by obtaining the target image description information input by the target user, input it into the text recognition network of the image query model, obtaining the target image tag, and selecting the reference image tag matching the target image tag from each candidate image tag, and finally outputting the candidate image corresponding to the reference image tag.
This method can quickly output images corresponding to the target image description information without manual participation, significantly improving the image query efficiency.
Smart Images
Figure CN119577167A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image scene recognition method, apparatus, computer equipment and storage medium. Background Art
[0002] As more and more image data are generated in people's daily lives, the storage space of hardware storage devices such as personal computers and mobile phones has become limited. People can choose to store image data in cloud disks, which can be accessed and shared anytime and anywhere. When accessing image data stored in cloud disks, people hope to quickly query the required images.
[0003] The image processing algorithms used for image query in traditional technologies, such as image enhancement and image compression, require manual image feature extraction, resulting in low image query efficiency. Summary of the invention
[0004] Based on this, it is necessary to provide an image query method, apparatus, computer equipment, storage medium and program product to improve the efficiency of image query in response to the above technical problems.
[0005] In a first aspect, the present application provides an image query method, the method comprising:
[0006] In response to the image query request, obtaining target image description information input by the target user;
[0007] Inputting the target image description information into the text recognition network of the image query model to obtain the target image label;
[0008] Selecting a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model;
[0009] Output candidate images in the image set corresponding to the reference image label.
[0010] In one embodiment, the image query model further includes an image preprocessing network and an image classification network; each candidate image label is determined by:
[0011] For each candidate image in the image set, input the candidate image into the image preprocessing network to obtain text description information corresponding to the candidate image;
[0012] Inputting the candidate image into the image classification network to obtain a first image label corresponding to the candidate image;
[0013] Inputting text description information corresponding to the candidate image into the text recognition network to obtain a second image label corresponding to the candidate image;
[0014] The first image label and the second image label are merged to obtain a candidate image label corresponding to the candidate image.
[0015] In one of the embodiments, the image classification network includes an image feature extraction layer and an image classification layer;
[0016] The step of inputting the candidate image into the image classification network to obtain a first image label corresponding to the candidate image includes:
[0017] Inputting the candidate image into the image feature extraction layer to obtain candidate image features corresponding to the candidate image;
[0018] The candidate image features are input into the image classification layer to obtain a first image label corresponding to the candidate image.
[0019] In one of the embodiments, the image feature extraction layer includes at least two sequentially connected image feature extraction units, each of which includes a base layer, a first residual module, an attention module, and a second residual module;
[0020] Among them, the base layer of each image feature extraction unit is used to extract features from input data to obtain basic image features; the first residual module is used to perform residual processing on the basic image features to obtain first residual image features; the attention module is used to adjust feature weights of the first residual image features to obtain adjusted image features; the second residual module is used to perform residual processing on the adjusted image features to obtain second residual image features;
[0021] The input data of the base layer of the first image feature extraction unit is the candidate image, and the input data of the base layer of any other image feature extraction unit is the second residual image feature output by the previous image feature extraction unit corresponding to the image feature extraction unit;
[0022] The second residual image feature output by the second residual module of the final image feature extraction unit is the candidate image feature corresponding to the candidate image.
[0023] In one of the embodiments, the text recognition network includes an encoding layer, a convolutional neural network layer, a continuous bag of words model layer, and a label prediction layer;
[0024] The step of inputting the text description information corresponding to the candidate image into the text recognition network to obtain a second image label corresponding to the candidate image includes:
[0025] Inputting text description information corresponding to the candidate image into the encoding layer to obtain basic encoding features;
[0026] Inputting the basic coding feature into the convolutional neural network layer to obtain a first deep image feature;
[0027] Inputting the first depth image feature into the continuous bag-of-words model layer to obtain a second depth image feature;
[0028] The first depth image feature and the second depth image feature are input into the label prediction layer to obtain a second image label corresponding to the candidate image.
[0029] In one embodiment, the candidate images in the image set are obtained by standardizing the initial images in the image set.
[0030] In a second aspect, the present application further provides an image query device, the device comprising:
[0031] An acquisition module, used for acquiring target image description information input by a target user in response to an image query request;
[0032] A recognition module, used for inputting the target image description information into a text recognition network of an image query model to obtain a target image label;
[0033] A selection module is used to select a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model;
[0034] An output module is used to output candidate images in the image set corresponding to the reference image label.
[0035] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:
[0036] In response to the image query request, obtaining target image description information input by the target user;
[0037] Inputting the target image description information into the text recognition network of the image query model to obtain the target image label;
[0038] Selecting a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model;
[0039] Output candidate images in the image set corresponding to the reference image label.
[0040] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0041] In response to the image query request, obtaining target image description information input by the target user;
[0042] Inputting the target image description information into the text recognition network of the image query model to obtain the target image label;
[0043] Selecting a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model;
[0044] Output candidate images in the image set corresponding to the reference image label.
[0045] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, which implements the following steps when executed by a processor:
[0046] In response to the image query request, obtaining target image description information input by the target user;
[0047] Inputting the target image description information into the text recognition network of the image query model to obtain the target image label;
[0048] Selecting a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model;
[0049] Output candidate images in the image set corresponding to the reference image label.
[0050] The above-mentioned image query method, apparatus, computer equipment, storage medium and program product, in response to an image query request, obtains the target image description information input by the target user; inputs the target image description information into the text recognition network of the image query model to obtain the target image label; and selects a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model; and then outputs the candidate image corresponding to the reference image label in the image set. The above-mentioned scheme, when there is an image query request, can input the target image description information into the image query model, and can quickly output the image corresponding to the target image description information without manual intervention, thereby improving the efficiency of image query. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 is a schematic diagram of a flow chart of an image query method in one embodiment;
[0052] Figure 2A is a schematic diagram of the structure of an image query model in one embodiment;
[0053] Figure 2B A schematic diagram of a process for determining labels of candidate images in one embodiment;
[0054] Figure 3A A schematic diagram of the structure of an image classification network in one embodiment;
[0055] Figure 3B A schematic diagram of a process of obtaining a first image label corresponding to a candidate image in one embodiment;
[0056] Figure 4 is a schematic diagram of the structure of an image feature extraction layer in one embodiment;
[0057] Figure 5A A schematic diagram of the structure of a text recognition network in one embodiment;
[0058] Figure 5B A schematic diagram of a process of obtaining a second image label corresponding to a candidate image in one embodiment;
[0059] Figure 6 is a structural block diagram of an image query device in one embodiment;
[0060] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] The image query method provided in the embodiment of the present application can be applied to the application scenario of querying images stored in the cloud according to image description information. The method can be executed by a server, specifically by a cloud server. The server can be implemented as an independent server or a server cluster composed of multiple servers.
[0063] In one embodiment, Figure 1 As shown, an image query method is provided, which is described by taking the method applied to a server as an example, and includes the following steps:
[0064] S101, in response to an image query request, obtaining target image description information input by a target user.
[0065] Exemplarily, the target user is a user who stores image data in the cloud server. When the target user has an image query request, for example, if he wants to query an image containing a cat stored in the server, he can enter the target image description information, for example, the target image description information can be "an image containing a cat"; the server obtains the target image description information entered by the target user.
[0066] S102, input the target image description information into the text recognition network of the image query model to obtain the target image label.
[0067] Furthermore, the image query model is a pre-trained model for querying images; the target image label can be understood as the image label corresponding to the target description information obtained after the target description information is processed by the text recognition model. The image label can be the category label of the image, such as animals, people and scenery, etc., or it can be a more specific animal category, person name and scenic spot or building name.
[0068] The target image query model can be deployed on the server or on other devices. If deployed on the server, the target image description information can be directly input into the text recognition network of the image query model to obtain the target image label. If deployed on other devices, the server needs to interact with other devices. For example, the server inputs the target image description information into the text recognition network of the image query model in other devices, and the other devices then transmit the obtained target image label back to the server.
[0069] S103: Select a reference image label that matches the target image label from each candidate image label.
[0070] Among them, each candidate image label is obtained by processing the image set input by the target user based on the image query model; for example, when the target user uploads each candidate image in the image set to the server, the server will input each candidate image into the image query model to obtain the candidate image label corresponding to each candidate image.
[0071] Based on this, a reference image label that matches the target image label can be selected from each candidate image label. For example, the target image label can be compared with each candidate image label for similarity, and the candidate image label with a similarity exceeding a preset threshold or the largest similarity can be used as the reference image label that matches the target image label.
[0072] S104, outputting candidate images in the image set corresponding to the reference image labels.
[0073] Furthermore, the candidate image corresponding to the reference image label can be output.
[0074] The above-mentioned image query method, in response to an image query request, obtains the target image description information input by the target user; inputs the target image description information into the text recognition network of the image query model to obtain the target image label; and selects a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model; and then outputs the candidate image corresponding to the reference image label in the image set. The above-mentioned scheme, when there is an image query request, can input the target image description information into the image query model, and can quickly output the image corresponding to the target image description information without manual intervention, thereby improving the efficiency of image query.
[0075] In some optional implementations, see Figure 2A , Figure 2A A structural diagram of an image query model is provided. Figure 2A It is shown that the image query model includes an image preprocessing network, a text recognition network, and an image classification network. Based on this, see Figure 2B , Figure 2B A schematic diagram of a process for determining labels of candidate images is provided, which specifically includes the following steps:
[0076] S201, for each candidate image in the image set, input the candidate image into an image preprocessing network to obtain text description information corresponding to the candidate image.
[0077] Exemplarily, for each candidate image in the image set, the candidate image can be input into the image preprocessing network to obtain text description information corresponding to the candidate image. For example, the image preprocessing network is used to obtain text description information corresponding to the candidate image, and a generative pre-trained chat model (Chat Generative Pre-trained Transformer, ChatGPT) can be used. The candidate image can be input into the image preprocessing network, and the instruction information "please add text description information for the candidate image" can be input to obtain the text description information corresponding to the candidate image.
[0078] S202, input the candidate image into an image classification network to obtain a first image label corresponding to the candidate image.
[0079] Furthermore, the candidate image can be input into the image classification network, which extracts features from the candidate image and predicts image labels based on the extracted features, and uses the label with the highest probability among the predicted image labels as the first image label. The image classification network is a network structure for outputting image labels based on an image. The first image label is the image label output by the image classification network based on the candidate image.
[0080] S203: Input text description information corresponding to the candidate image into a text recognition network to obtain a second image label corresponding to the candidate image.
[0081] Furthermore, the text description information corresponding to the candidate image can also be input into the text recognition network, which extracts features from the text description information and predicts image labels based on the extracted features, and uses the image label with the highest probability among the predicted image labels as the second image label.
[0082] S204: Fusing the first image label and the second image label to obtain a candidate image label corresponding to the candidate image.
[0083] Furthermore, the first image label and the second image label may be fused to improve the accuracy of image label prediction, thereby obtaining a candidate image label corresponding to the candidate image.
[0084] In an embodiment of the present application, in the process of generating candidate image labels corresponding to each candidate image, an image classification network and a text recognition network are used to generate image labels respectively, and the image labels generated by the two are fused to obtain the final candidate image labels, so that the final generated candidate image labels incorporate more dimensional features, thereby improving the accuracy of generating candidate image labels.
[0085] For example, see Figure 3A , Figure 3AA structural diagram of an image classification network is provided. Figure 3A It is shown that the image classification network in the above embodiment may include an image feature extraction layer and an image classification layer.
[0086] Based on this, see Figure 3B , Figure 3B A schematic diagram of a process for obtaining a first image label corresponding to a candidate image is provided, which specifically includes the following steps:
[0087] S301, inputting the candidate image into the image feature extraction layer to obtain the candidate image features corresponding to the candidate image.
[0088] Exemplarily, the candidate image may be input into an image feature extraction layer to perform feature extraction to obtain candidate image features corresponding to the candidate image.
[0089] S302: Input the candidate image features into the image classification layer to obtain a first image label corresponding to the candidate image.
[0090] Furthermore, the candidate image features can be input into the image classification layer, and the image classification layer can predict the image label based on the candidate image features by calling the activation function, and use the image label with the highest probability among the predicted image labels as the first image label. The activation function can be a normalized exponential function softmax.
[0091] In an embodiment of the present application, features of the candidate image are extracted through the image feature extraction layer, and the image classification layer obtains the first image label corresponding to the candidate image based on the candidate image features obtained by the image feature extraction layer, thereby realizing feature extraction from the candidate image dimension and obtaining the first image label, which facilitates the subsequent generation of candidate image labels for the candidate image.
[0092] Further, see Figure 4 , Figure 4 A structural diagram of an image feature extraction layer is provided. Figure 4 It is shown that the image feature extraction layer includes at least two image feature extraction units connected in sequence. In the embodiment of the present application, the image feature extraction layer includes three image feature extraction units connected in sequence. Among them, the image feature extraction units can be connected using a conversion layer. The conversion layer convolves the second residual image feature output by the previous image feature extraction unit through a large-step convolution kernel, which can adjust the size and number of channels of the second residual image feature, and complete the step-by-step reduction of the input size requirements and the number of parameters of the lower-level image feature extraction unit. The image feature extraction unit adopts a combination of the attention module and the residual network ResNet structure. Each image feature extraction unit includes a base layer, a first residual module, an attention module, and a second residual module.
[0093] Among them, the base layer can be a convolution layer and a pooling layer connected in sequence; the convolution layer performs local operations on the input data through the convolution kernel, and the convolution kernel slides on the input data (called stride) to calculate the weighted sum of each local area, thereby generating new image features. This local feature extraction method can effectively reduce the number of parameters and improve computational efficiency. The pooling layer reduces the complexity of the model by reducing the spatial resolution of the input, which helps prevent overfitting.
[0094] The first residual module and the second residual module are used to solve the gradient vanishing and gradient exploding problems and improve the accuracy and stability of the model. The core of the first residual module and the second residual module is the residual connection, which adds the input data to the output data to form a cross-layer connection. This connection method allows the error to propagate from the output layer back to the previous layer, making it easier for the model to learn residual information. In addition, the residual connection also helps to retain the characteristics of the input data and avoid losing important information in the deep network.
[0095] The attention module can be a Squeeze (squeeze / compression) and Excitation (excitation) SE module of the Squeeze-and-Excitation Networks (SENet), which is used to measure the network's attention to features in different regions, and to increase the weights of certain key features so that the network pays more attention to features that are critical to the task. The attention module can dynamically adjust the weights of feature maps, thereby enhancing the model's sensitivity to key features, which helps to improve the recognition accuracy and performance of the model. For example, the attention module uses global average pooling (GAP) to aggregate the channel information of the feature map, and uses a linear rectifier function (Rectified Linear Unit, ReLU) to learn the importance weights of the channels, and multiplies the weights by the first residual image features to obtain adjusted image features.
[0096] Exemplarily, the base layer of each image feature extraction unit is used to perform feature extraction on the input data to obtain basic image features; the first residual module is used to perform residual processing on the basic image features to obtain first residual image features; the attention module is used to adjust the feature weights of the first residual image features to obtain adjusted image features; the second residual module is used to perform residual processing on the adjusted image features to obtain second residual image features.
[0097] The input data of the base layer of the first image feature extraction unit is the candidate image, and the input data of the base layer of any other image feature extraction unit is the second residual image feature output by the previous image feature extraction unit corresponding to the image feature extraction unit.
[0098] The second residual image feature output by the second residual module of the final image feature extraction unit is the candidate image feature corresponding to the candidate image.
[0099] In an embodiment of the present application, by setting at least two sequentially connected image feature extraction units, and setting each image feature extraction unit to include a base layer, a first residual module, an attention module, and a second residual module, each image feature extraction unit can obtain more dimensional feature data, so that the candidate image features corresponding to the determined candidate images have more dimensional feature data, and the image classification layer can determine a more accurate first image label based on the more dimensional candidate image features.
[0100] In some optional implementations, see Figure 5A , Figure 5A A structural diagram of a text recognition network is provided. Figure 5A It shows that the text recognition network includes an encoding layer, a convolutional neural network layer, a continuous bag of words model layer, and a label prediction layer.
[0101] Based on this, see Figure 5B , Figure 5B A schematic diagram of a process for obtaining a second image label corresponding to a candidate image is provided, which specifically includes the following steps:
[0102] S501, input text description information corresponding to the candidate image into the encoding layer to obtain basic encoding features.
[0103] The encoding layer can adopt a pre-trained continuous bag-of-words (CBOW) model. For example, the text description information is input into the encoding layer, and the encoding layer converts the text description information into a one-hot encoding of a fixed-dimensional word vector sequence, and uses the obtained one-hot encoding as the basic encoding feature.
[0104] S502, input the basic coding features into the convolutional neural network layer to obtain the first deep image features.
[0105] Among them, the convolutional neural network layer can adopt a convolutional neural network (CNN), and CNN can be composed of a convolution layer composed of a one-dimensional convolution kernel, an activation function layer, and a pooling layer connected in sequence. The basic coding features are further extracted through CNN to obtain deeper image features, that is, the first deep image features. The activation function layer in CNN can adopt a linear rectified unit (ReLU).
[0106] S503: Input the first depth image feature into a continuous bag-of-words model layer to obtain a second depth image feature.
[0107] Among them, the continuous bag-of-words model layer (CBOW) converts the basic encoding features of the input into word vectors to obtain deeper image features, namely the second deep image features.
[0108] S504: Input the first depth image feature and the second depth image feature into a label prediction layer to obtain a second image label corresponding to the candidate image.
[0109] The label prediction layer can be a fully connected layer and an activation function layer. The fully connected layer will fuse the first depth image features and the second depth image features, and further extract features. The image features output by the fully connected layer are input into the activation function layer to predict the image label and obtain the second image label corresponding to the candidate image. The activation function layer in the label prediction layer can use softmax.
[0110] In an embodiment of the present application, the text recognition network is set to an encoding layer, a convolutional neural network layer, a continuous bag-of-words model layer and a label prediction layer, which can perform layer-by-layer feature extraction on the candidate image, and obtain image features of more dimensions of the candidate image, thereby making the second image label corresponding to the determined candidate image more accurate.
[0111] In some optional implementations, in order to facilitate the image query model to process candidate images and improve the accuracy and efficiency of outputting candidate images, the image set uploaded by the target user can be standardized, that is, the candidate images in the image set are obtained by standardizing the initial images in the image set. For example, the initial images in the image set can be first adjusted to the same size, and then the L1 regularization operation can be performed on the images adjusted to the same size to remove noise in the image data, so that the image data presents a normal distribution, which is convenient for the image query model to process the candidate images and avoids overfitting of the image query model.
[0112] In an embodiment of the present application, the initial images in the image set are standardized to obtain candidate images in the image set, which improves the rate at which the image query model processes candidate images and avoids overfitting of the image query model, thereby improving the accuracy of the output candidate images.
[0113] In some optional implementations, the image query model is trained in the following manner:
[0114] Step 1: Preprocessing of sample image set.
[0115] Exemplarily, a sample image set may be obtained from a server, wherein the sample image set includes original images I1, ..., I n ; Original image I1,…,I n The corresponding sample image labels are L1,…,L n Then the original images I1,…,I n For example, the initial images in the image set can be first adjusted to the same size, and then the images adjusted to the same size can be subjected to L1 regularization to obtain the sample images S1,…,S n . Where n is the number of original images.
[0116] Step 2: Use sample images to train the initial model.
[0117] First, the image preprocessing network ChatGPT in the initial model can be used to generate text description information C1,…,C n The image in the input image preprocessing network ChatGPT can be the original image or the standardized image S1,…,S n .
[0118] Furthermore, each sample image S1,…,S n Input it into the image classification network in the initial model to obtain the predicted first image label corresponding to each sample image; and according to the first image label corresponding to each sample image and each sample image label L1,…,L n Determine the first loss L ce .
[0119] And the text description information C1,…,C n Input into the text recognition network in the initial model to obtain the predicted second image label corresponding to each sample image; and according to the second image label corresponding to each sample image and each sample image label L1,…,L n Determine the second loss L ll .
[0120] Then determine the comprehensive loss of the initial model:
[0121]
[0122] in, is the comprehensive loss of the initial model, is the weight coefficient of the first loss, is the weight coefficient of the second loss.
[0123] Among them, the first loss L ce Determined by:
[0124]
[0125] Among them, m is the total number of samples, a is the number of categories to which the samples belong, is the label of category j to which sample i belongs, To predict the probability that sample i belongs to category j; is the sample i belonging to category j.
[0126] Among them, the second loss L ll Determined by:
[0127]
[0128] Among them, m is the total number of samples, is the label of sample i, is the probability of the category to which the predicted sample i belongs.
[0129] in and It can be adjusted dynamically according to the actual scenario. The process of optimizing the initial model is a process in which the output of the initial model is closer and closer to the training label data set, and the output is more accurate. ce and L ll The joint supervision optimization network of Gradually tends to a stable effective value, so that the prediction output of the initial model gradually approaches the true label.
[0130] In the embodiment of the present application, real-time dynamic new cloud image scene understanding can be realized. Through the automated image understanding and retrieval process, the user's time and energy investment in image management and access can be reduced, providing more convenient and personalized services.
[0131] The image query model includes an image classification network and a text recognition network, and a joint loss is used in the training of the image query model to effectively improve the network's scene understanding and recognition capabilities. In addition, deep learning and graphics processing units (GPUs) are used to accelerate computing, which can quickly process and analyze large amounts of image data, achieve fast retrieval, and improve user experience.
[0132] At the same time, the image query model can adapt to the growing amount of image data and has good scalability. Users can quickly find the required images by simply entering keywords, which simplifies the user operation process. The use of ChatGPT technology to automatically generate text descriptions of images reduces the dependence on professional image processing knowledge. By comprehensively utilizing multiple features and deep learning models, the image query model is more adaptable and robust to different scenarios and conditions.
[0133] Furthermore, the embodiments of the present application can perform scene understanding, recognition and storage of the user's cloud image data, and achieve real-time response to user retrieval. Accurate image understanding and retrieval capabilities lay the foundation for providing personalized services and recommending content based on user preferences and behavior patterns. Efficient image retrieval is achieved with minimal resource investment, optimizing the use of computing resources and storage resources.
[0134] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0135] Based on the same inventive concept, the embodiment of the present application also provides an image query device for implementing the above-mentioned image query method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image query device embodiments provided below can refer to the limitations on the image query method above, and will not be repeated here.
[0136] In one embodiment, Figure 6 As shown, an image query device is provided, comprising:
[0137] An acquisition module 10 is used to acquire target image description information input by a target user in response to an image query request;
[0138] The recognition module 20 is used to input the target image description information into the text recognition network of the image query model to obtain the target image label;
[0139] A selection module 30 is used to select a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model;
[0140] The output module 40 is used to output candidate images in the image set corresponding to the reference image labels.
[0141] The above-mentioned image query device, in response to an image query request, obtains the target image description information input by the target user; inputs the target image description information into the text recognition network of the image query model to obtain the target image label; and selects a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model; and then outputs the candidate image corresponding to the reference image label in the image set. The above-mentioned scheme, when there is an image query request, can input the target image description information into the image query model, and can quickly output the image corresponding to the target image description information without manual intervention, thereby improving the efficiency of image query.
[0142] In one of the embodiments, the device further comprises a pre-processing module and a classification module;
[0143] A preprocessing module is used to input each candidate image in the image set into an image preprocessing network to obtain text description information corresponding to the candidate image;
[0144] A classification module, used to input the candidate image into an image classification network to obtain a first image label corresponding to the candidate image;
[0145] The recognition module 20 is further used to input the text description information corresponding to the candidate image into the text recognition network to obtain the second image label corresponding to the candidate image;
[0146] The fusion module is used to fuse the first image label and the second image label to obtain a candidate image label corresponding to the candidate image.
[0147] In one embodiment, the image classification network includes an image feature extraction layer and an image classification layer; the classification module includes:
[0148] A feature extraction unit, used to input the candidate image into the image feature extraction layer to obtain candidate image features corresponding to the candidate image;
[0149] The classification unit is used to input the candidate image features into the image classification layer to obtain the first image label corresponding to the candidate image.
[0150] In one of the embodiments, the image feature extraction layer includes at least two sequentially connected image feature extraction units, each of which includes a base layer, a first residual module, an attention module, and a second residual module;
[0151] Among them, the base layer of each image feature extraction unit is used to extract features from input data to obtain basic image features; the first residual module is used to perform residual processing on the basic image features to obtain first residual image features; the attention module is used to adjust the feature weights of the first residual image features to obtain adjusted image features; the second residual module is used to perform residual processing on the adjusted image features to obtain second residual image features;
[0152] The input data of the base layer of the first image feature extraction unit is the candidate image, and the input data of the base layer of any other image feature extraction unit is the second residual image feature output by the previous image feature extraction unit corresponding to the image feature extraction unit;
[0153] The second residual image feature output by the second residual module of the final image feature extraction unit is the candidate image feature corresponding to the candidate image.
[0154] In one embodiment, the text recognition network includes an encoding layer, a convolutional neural network layer, a continuous bag-of-words model layer, and a label prediction layer; the recognition module 20 is specifically used for:
[0155] The text description information corresponding to the candidate image is input into the encoding layer to obtain the basic encoding features; the basic encoding features are input into the convolutional neural network layer to obtain the first deep image features; the first deep image features are input into the continuous bag of words model layer to obtain the second deep image features; the first deep image features and the second deep image features are input into the label prediction layer to obtain the second image label corresponding to the candidate image.
[0156] In one embodiment, the candidate images in the image set are obtained by standardizing the initial images in the image set.
[0157] Each module in the above-mentioned image query device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.
[0158] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image query method is implemented.
[0159] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0160] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0161] In response to the image query request, obtaining target image description information input by the target user;
[0162] Input the target image description information into the text recognition network of the image query model to obtain the target image label;
[0163] Selecting a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing an image set input by a target user based on an image query model;
[0164] Output candidate images in the image set that correspond to the reference image labels.
[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0166] For each candidate image in the image set, the candidate image is input into the image preprocessing network to obtain the text description information corresponding to the candidate image; the candidate image is input into the image classification network to obtain the first image label corresponding to the candidate image; the text description information corresponding to the candidate image is input into the text recognition network to obtain the second image label corresponding to the candidate image; the first image label and the second image label are fused to obtain the candidate image label corresponding to the candidate image.
[0167] In one embodiment, the image classification network includes an image feature extraction layer and an image classification layer; when the processor executes the computer program, the following steps are also implemented:
[0168] The candidate image is input into the image feature extraction layer to obtain the candidate image features corresponding to the candidate image; the candidate image features are input into the image classification layer to obtain the first image label corresponding to the candidate image.
[0169] In one embodiment, the image feature extraction layer includes at least two image feature extraction units connected in sequence, each image feature extraction unit includes a base layer, a first residual module, an attention module and a second residual module; wherein the base layer of each image feature extraction unit is used to perform feature extraction on the input data to obtain a basic image feature; the first residual module is used to perform residual processing on the basic image feature to obtain a first residual image feature; the attention module is used to adjust the feature weight of the first residual image feature to obtain an adjusted image feature; the second residual module is used to perform residual processing on the adjusted image feature to obtain a second residual image feature; the input data of the base layer of the first image feature extraction unit is the candidate image, and the input data of the base layer of any other image feature extraction unit is the second residual image feature output by the previous image feature extraction unit corresponding to the image feature extraction unit; the second residual image feature output by the second residual module of the last image feature extraction unit is the candidate image feature corresponding to the candidate image.
[0170] In one embodiment, the text recognition network includes an encoding layer, a convolutional neural network layer, a continuous bag of words model layer, and a label prediction layer; when the processor executes the computer program, the following steps are also implemented:
[0171] Input the text description information corresponding to the candidate image into the encoding layer to obtain the basic encoding features; input the basic encoding features into the convolutional neural network layer to obtain the first deep image features;
[0172] The first deep image feature is input into the continuous bag-of-words model layer to obtain the second deep image feature; the first deep image feature and the second deep image feature are input into the label prediction layer to obtain the second image label corresponding to the candidate image.
[0173] In one embodiment, the candidate images in the image set are obtained by standardizing the initial images in the image set.
[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image query method described in any of the above embodiments are implemented.
[0175] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the image query method described in any embodiment are implemented.
[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0179] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An image query method, characterized in that: The method comprises: In response to the image query request, obtaining target image description information input by the target user; Inputting the target image description information into the text recognition network of the image query model to obtain the target image label; Selecting a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model; Output candidate images in the image set corresponding to the reference image label.
2. The method according to claim 1, characterized in that The image query model also includes an image preprocessing network and an image classification network; each candidate image label is determined by: For each candidate image in the image set, input the candidate image into the image preprocessing network to obtain text description information corresponding to the candidate image; Inputting the candidate image into the image classification network to obtain a first image label corresponding to the candidate image; Inputting text description information corresponding to the candidate image into the text recognition network to obtain a second image label corresponding to the candidate image; The first image label and the second image label are merged to obtain a candidate image label corresponding to the candidate image.
3. The method according to claim 2, characterized in that The image classification network includes an image feature extraction layer and an image classification layer; The step of inputting the candidate image into the image classification network to obtain a first image label corresponding to the candidate image includes: Inputting the candidate image into the image feature extraction layer to obtain candidate image features corresponding to the candidate image; The candidate image features are input into the image classification layer to obtain a first image label corresponding to the candidate image.
4. The method according to claim 3, characterized in that The image feature extraction layer includes at least two sequentially connected image feature extraction units, each of which includes a base layer, a first residual module, an attention module, and a second residual module; Among them, the base layer of each image feature extraction unit is used to extract features from input data to obtain basic image features; the first residual module is used to perform residual processing on the basic image features to obtain first residual image features; the attention module is used to adjust feature weights of the first residual image features to obtain adjusted image features; the second residual module is used to perform residual processing on the adjusted image features to obtain second residual image features; The input data of the base layer of the first image feature extraction unit is the candidate image, and the input data of the base layer of any other image feature extraction unit is the second residual image feature output by the previous image feature extraction unit corresponding to the image feature extraction unit; The second residual image feature output by the second residual module of the final image feature extraction unit is the candidate image feature corresponding to the candidate image.
5. The method according to claim 2, characterized in that: The text recognition network includes an encoding layer, a convolutional neural network layer, a continuous bag of words model layer and a label prediction layer; The step of inputting the text description information corresponding to the candidate image into the text recognition network to obtain a second image label corresponding to the candidate image includes: Inputting text description information corresponding to the candidate image into the encoding layer to obtain basic encoding features; Inputting the basic coding feature into the convolutional neural network layer to obtain a first deep image feature; Inputting the first depth image feature into the continuous bag-of-words model layer to obtain a second depth image feature; The first depth image feature and the second depth image feature are input into the label prediction layer to obtain a second image label corresponding to the candidate image.
6. The method according to any one of claims 1 to 5, characterized in that: The candidate images in the image set are obtained by standardizing the initial images in the image set.
7. An image query device, characterized in that: The device comprises: An acquisition module, used for acquiring target image description information input by a target user in response to an image query request; A recognition module, used for inputting the target image description information into a text recognition network of an image query model to obtain a target image label; A selection module is used to select a reference image label that matches the target image label from each candidate image label; wherein each candidate image label is obtained by processing the image set input by the target user based on the image query model; An output module is used to output candidate images in the image set corresponding to the reference image label.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.