Image feature acquisition method, device, computer equipment, and storage medium

Abstract

The present application relates to an image feature acquisition method, device, computer equipment, storage medium and computer program product. The method relates to computer vision technology of artificial intelligence, and the method comprises: upon receiving a feature query request carrying a target image identifier sent by a business party, returning a general image feature of the target image obtained by extracting features from a target image corresponding to the target image identifier by calling a general image feature extraction model. The target image identifier and the general image feature of the target image are stored in a general image feature library in correspondence, so that when a feature query request carrying the target image identifier sent by any business party is received again, the general image feature corresponding to the target image identifier is directly returned, thereby reducing computing consumption.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, storage medium and computer program product for acquiring image features, and also to a method, apparatus, computer device, storage medium and computer program product for training a general image feature extraction model. Background Art

[0002] With the rapid development of computer technology and artificial intelligence, computer vision has been widely used. Computer vision tasks enable computers to understand the content of pictures or videos, thereby realizing a series of business scenarios, such as image classification, image retrieval, etc. Image features play a vital role in computer vision tasks. They are the representation of the characteristics or content of an image, and are the basic information that distinguishes one image from another, and are used to identify and mark image content.

[0003] In traditional technology, for the same image or video, in different business scenarios, the same image or video is processed by an image feature extraction model suitable for the corresponding business scenario to obtain image features suitable for the corresponding business scenario, and then the extracted image features are applied to the corresponding business (such as image classification, image retrieval). In other words, the same image or video will be processed separately by image feature extraction models suitable for different businesses. These image feature extraction models suitable for different businesses often have many parameters and large amount of calculation, and will consume a lot of computing resources in the training and prediction stages. Summary of the invention

[0004] Based on this, it is necessary to provide an image feature acquisition method, apparatus, computer equipment, computer-readable storage medium and computer program product that can reduce computing consumption in order to solve the above technical problems.

[0005] The present application provides a method for acquiring image features. The method comprises:

[0006] Upon receiving a feature query request carrying a target image identifier sent by a business party, return a general image feature of the target image obtained by calling a general image feature extraction model to extract features from the target image corresponding to the target image identifier; wherein the general image feature extraction model is obtained by jointly training an image feature extraction model and a text feature extraction model, each training sample includes a sample image and text information of the sample image, the image feature extraction model is updated according to a matching loss function determined based on a plurality of the training samples, and the matching loss function is determined according to the degree of matching between the image feature obtained by extracting features from the sample image using the image feature extraction model and the text feature obtained by extracting features from the text information using the text feature extraction model;

[0007] The target image identifier and the universal image features of the target image are stored in a universal image feature library in correspondence, so that when a feature query request carrying the target image identifier is received again from any business party, the universal image features corresponding to the target image identifier are directly returned.

[0008] The present application also provides an image feature acquisition device. The device comprises:

[0009] A receiving module, used for returning the general image features of the target image obtained by extracting features from the target image corresponding to the target image identifier by calling a general image feature extraction model upon receiving a feature query request carrying a target image identifier sent by a business party; wherein the general image feature extraction model is obtained by jointly training an image feature extraction model and a text feature extraction model, each training sample includes a sample image and text information of the sample image, the image feature extraction model is updated according to a matching loss function determined based on a plurality of the training samples, and the matching loss function is determined according to the degree of matching between the image features obtained by extracting features from the sample image by the image feature extraction model and the text features obtained by extracting features from the text information by the text feature extraction model;

[0010] The storage module is used to store the target image identifier and the universal image features of the target image in a universal image feature library so that when a feature query request carrying the target image identifier is received again from any business party, the universal image features corresponding to the target image identifier are directly returned.

[0011] In one embodiment, the receiving module is used to, when receiving a feature query request carrying a target image identifier sent by a business party, obtain a target image corresponding to the target image identifier based on the target image identifier, and call a universal image feature extraction model to perform feature extraction on the target image to obtain a universal image feature corresponding to the target image identifier, and return the universal image feature corresponding to the target image identifier in response to the feature query request.

[0012] In one embodiment, the receiving module is used to query whether there is a universal image feature corresponding to the target image identifier in the universal image feature library based on the mapping relationship between the image identifier and the universal image feature when a feature query request carrying the target image identifier is received again from any business party; if it is found that the universal image feature corresponding to the target image identifier exists in the universal image feature library, then in response to the feature query request, the universal image feature corresponding to the target image identifier queried from the universal image feature library is returned; if it is found that the universal image feature corresponding to the target image identifier does not exist in the universal image feature library, then in response to the feature query request, an abnormal prompt message is returned.

[0013] In one embodiment, the image feature acquisition device further includes a clearing module, and the clearing module is used to clear the general image features stored in the general image feature library when an updated general image feature extraction model is received.

[0014] In one embodiment, the receiving module is also used to return the updated universal image features of the target image obtained by calling the updated universal image feature extraction model to extract features from the target image corresponding to the target image identifier if a feature query request carrying the target image identifier is received from a business party; the storage module is also used to store the target image identifier and the updated universal image features of the target image in a universal image feature library in correspondence with each other, so as to directly return the updated universal image features corresponding to the target image identifier when a feature query request carrying the target image identifier is received from any business party again.

[0015] In one embodiment, the image feature acquisition device also includes a training module, which is used to acquire multiple training samples, each of which includes a sample image and text information corresponding to the sample image; through an image feature extraction model, feature extraction is performed on the sample image of each training sample to obtain the image feature of each training sample; through a text feature extraction model, feature extraction is performed on the text information of each training sample to obtain the text feature of each training sample; a positive example matching loss is constructed based on the image features and text features of the same training sample; a negative example matching loss is constructed based on the image features and text features of each training sample; a matching loss function is determined based on the positive example matching loss and the negative example matching loss; according to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained, and the trained image feature extraction model is used as a general image feature extraction model.

[0016] In one embodiment, the training module is used to calculate the similarity between the image feature and the text feature of each training sample, obtain the intra-sample feature matching degree of each training sample, and determine the positive example matching loss according to the intra-sample feature matching degree of each training sample; the training module is used to calculate the similarity between the image feature of the training sample and the text feature of each training sample other than the training sample, obtain the inter-sample feature matching degree of each training sample, and determine the negative example matching loss according to the inter-sample feature matching degree of each training sample; the training module is used to determine the matching loss function according to the intra-sample feature matching degree of each training sample and the inter-sample feature matching degree, and the matching loss function is inversely correlated with the intra-sample feature matching degree and positively correlated with the inter-sample feature matching degree.

[0017] In one embodiment, the training module is used to calculate the sum of the cross entropies of the feature matching degrees within each sample, and calculate the sum of the cross entropies of the feature matching degrees between each sample; and determine the matching loss function based on the sum of the cross entropies of the feature matching degrees within each sample and the sum of the cross entropies of the feature matching degrees between each sample.

[0018] In one embodiment, the training module is used to update the network parameters of the image feature extraction model and the text feature extraction model with the goal of minimizing the matching loss function, and then return to the step of obtaining multiple training samples to continue training until the training stop condition is met, thereby obtaining a general image feature extraction model.

[0019] The present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0020] Upon receiving a feature query request carrying a target image identifier sent by a business party, return a general image feature of the target image obtained by calling a general image feature extraction model to extract features from the target image corresponding to the target image identifier; wherein the general image feature extraction model is obtained by jointly training an image feature extraction model and a text feature extraction model, each training sample includes a sample image and text information of the sample image, the image feature extraction model is updated according to a matching loss function determined based on a plurality of the training samples, and the matching loss function is determined according to the degree of matching between the image feature obtained by extracting features from the sample image using the image feature extraction model and the text feature obtained by extracting features from the text information using the text feature extraction model;

[0021] The target image identifier and the universal image features of the target image are stored in a universal image feature library in correspondence, so that when a feature query request carrying the target image identifier is received again from any business party, the universal image features corresponding to the target image identifier are directly returned.

[0022] The present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0023] Upon receiving a feature query request carrying a target image identifier sent by a business party, return a general image feature of the target image obtained by calling a general image feature extraction model to extract features from the target image corresponding to the target image identifier; wherein the general image feature extraction model is obtained by jointly training an image feature extraction model and a text feature extraction model, each training sample includes a sample image and text information of the sample image, the image feature extraction model is updated according to a matching loss function determined based on a plurality of the training samples, and the matching loss function is determined according to the degree of matching between the image feature obtained by extracting features from the sample image using the image feature extraction model and the text feature obtained by extracting features from the text information using the text feature extraction model;

[0024] The target image identifier and the universal image features of the target image are stored in a universal image feature library in correspondence, so that when a feature query request carrying the target image identifier is received again from any business party, the universal image features corresponding to the target image identifier are directly returned.

[0025] The present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Upon receiving a feature query request carrying a target image identifier sent by a business party, return a general image feature of the target image obtained by calling a general image feature extraction model to extract features from the target image corresponding to the target image identifier; wherein the general image feature extraction model is obtained by jointly training an image feature extraction model and a text feature extraction model, each of the training samples includes a sample image and text information of the sample image, the image feature extraction model is updated according to a matching loss function determined based on the multiple training samples, and the matching loss function is determined according to the degree of matching between the image feature obtained by extracting features from the sample image using the general image feature extraction model and the text feature obtained by extracting features from the text information using the text feature extraction model;

[0027] The target image identifier and the universal image features of the target image are stored in a universal image feature library in correspondence, so that when a feature query request carrying the target image identifier is received again from any business party, the universal image features corresponding to the target image identifier are directly returned.

[0028] The above-mentioned image feature acquisition method, device, computer equipment, storage medium and computer program product, since the universal image feature extraction model is obtained by jointly training the image feature extraction model and the text feature extraction model through multiple training samples, during the training process, the text information of the sample image is used as supervision information, and the supervision information of the image in the specific business scenario is not used, so that the trained image feature extraction model has the ability to mine the common information of the target images suitable for different business parties. Therefore, when a feature query request carrying the target image identifier sent by the business party is received, the universal image feature of the target image obtained by calling the universal image feature extraction model to extract the target image corresponding to the target image identifier is returned, and the target image identifier and the universal image feature of the target image are stored in the universal image feature library in correspondence. In this way, when a feature query request carrying the target image identifier sent by any business party is received again, the universal image feature corresponding to the target image identifier is directly returned, avoiding repeated calculations of the same image by different business parties, and reducing the computing consumption caused by repeated calculations.

[0029] The present application also provides a training method for a general image feature extraction model, the method comprising:

[0030] Acquire a plurality of training samples, each of the training samples comprising a sample image and text information corresponding to the sample image;

[0031] Extracting features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample;

[0032] By using a text feature extraction model, feature extraction is performed on the text information of each training sample to obtain text features of each training sample;

[0033] Construct positive example matching loss based on the image features and text features of the same training sample;

[0034] Construct negative example matching loss based on the image features and text features of each pair of training samples;

[0035] Determining a matching loss function according to the positive example matching loss and the negative example matching loss;

[0036] According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained, and the trained image feature extraction model is used as a general image feature extraction model.

[0037] The present application provides a training device for a general image feature extraction model. The device comprises:

[0038] An acquisition module, used for acquiring a plurality of training samples, each of which includes a sample image and text information corresponding to the sample image;

[0039] An extraction module is used to extract features from the sample images of each training sample through an image feature extraction model to obtain image features of each training sample; and to extract features from the text information of each training sample through a text feature extraction model to obtain text features of each training sample;

[0040] A construction module is used to construct a positive example matching loss based on the image features and text features of the same training sample; and to construct a negative example matching loss based on the image features and text features of each pair of training samples;

[0041] A determination module, configured to determine a matching loss function according to the positive example matching loss and the negative example matching loss;

[0042] An updating module is used to jointly train the image feature extraction model and the text feature extraction model according to the matching loss function, and use the trained image feature extraction model as a general image feature extraction model.

[0043] The present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0044] Acquire a plurality of training samples, each of the training samples comprising a sample image and text information corresponding to the sample image;

[0045] Extracting features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample;

[0046] By using a text feature extraction model, feature extraction is performed on the text information of each training sample to obtain text features of each training sample;

[0047] Construct positive example matching loss based on the image features and text features of the same training sample;

[0048] Construct negative example matching loss based on the image features and text features of each pair of training samples;

[0049] Determining a matching loss function according to the positive example matching loss and the negative example matching loss;

[0050] According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained, and the trained image feature extraction model is used as a general image feature extraction model.

[0051] The present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Acquire a plurality of training samples, each of the training samples comprising a sample image and text information corresponding to the sample image;

[0053] Extracting features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample;

[0054] By using a text feature extraction model, feature extraction is performed on the text information of each training sample to obtain text features of each training sample;

[0055] Construct positive example matching loss based on the image features and text features of the same training sample;

[0056] Construct negative example matching loss based on the image features and text features of each pair of training samples;

[0057] Determining a matching loss function according to the positive example matching loss and the negative example matching loss;

[0058] According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained, and the trained image feature extraction model is used as a general image feature extraction model.

[0059] The present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0060] Acquire a plurality of training samples, each of the training samples comprising a sample image and text information corresponding to the sample image;

[0061] Extracting features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample;

[0062] By using a text feature extraction model, feature extraction is performed on the text information of each training sample to obtain text features of each training sample;

[0063] Construct positive example matching loss based on the image features and text features of the same training sample;

[0064] Construct negative example matching loss based on the image features and text features of each pair of training samples;

[0065] Determining a matching loss function according to the positive example matching loss and the negative example matching loss;

[0066] According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained, and the trained image feature extraction model is used as a general image feature extraction model.

[0067] The training method, apparatus, computer equipment, storage medium and computer program product of the above-mentioned general image feature extraction model obtain multiple training samples, each of which includes a sample image and text information corresponding to the sample image. Through the image feature extraction model, feature extraction is performed on the sample image of each training sample to obtain the image feature of each training sample. Through the text feature extraction model, feature extraction is performed on the text information of each training sample to obtain the text feature of each training sample. In this way, each training sample corresponds to the image feature and text feature that match it. A positive example matching loss is constructed based on the image features and text features of the same training sample to reflect the gap between the matching image features and text features. A negative example matching loss is constructed based on the image features and text features of the pairwise training samples to reflect the difference between the unmatched image features and text features. A matching loss function is determined based on the positive example matching loss and the negative example matching loss. According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained. During the training process, the text information of the sample image is used as the supervisory information, and the supervisory information of the image in the specific business scenario is not used. In this way, the trained image feature extraction model has the ability to mine the common information of target images suitable for different business parties. In this way, when the trained image feature extraction model is used as a general image feature extraction model, it can accurately extract general image features suitable for any business. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1A A schematic diagram of a process of classifying an image or video in one embodiment;

[0069] Figure 1B A schematic diagram of a process flow of a multimodal classification service in one embodiment;

[0070] Figure 1C A schematic diagram of a process flow of a video frame retrieval service in one embodiment;

[0071] Figure 1DA schematic diagram of a process flow of a multimodal retrieval service in one embodiment;

[0072] Figure 2 is a schematic diagram of a flow chart of an image feature acquisition method in one embodiment;

[0073] Figure 3 A diagram showing an application environment of an image feature acquisition method in an embodiment;

[0074] Figure 4 is a flow chart of an image feature acquisition method in another embodiment;

[0075] Figure 5 A schematic diagram of a process for training a general image feature extraction model in one embodiment;

[0076] Figure 6 is a schematic diagram of application of universal image features in one embodiment;

[0077] Figure 7 A schematic diagram of a flow chart of the training steps of a general image feature extraction model in one embodiment;

[0078] Figure 8 A schematic diagram of a flow chart of the training steps of a general image feature extraction model in another embodiment;

[0079] Fig. 9 A schematic diagram of a flow chart of a training method for a general image feature extraction model in one embodiment;

[0080] Fig.10 is a flow chart of an image feature acquisition method in another embodiment;

[0081] Fig.11 is a structural block diagram of an image feature acquisition device in one embodiment;

[0082] Fig.12 is a structural block diagram of a training device for a general image feature extraction model in one embodiment;

[0083] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0085] The image feature acquisition method and the training method of the general image feature extraction model provided in the embodiments of the present application involve artificial intelligence (AI) technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0086] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The image feature acquisition method and the training method of the general image feature extraction model provided in the embodiments of the present application specifically relate to the computer vision technology of artificial intelligence technology.

[0087] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0088] In short, computer vision refers to enabling computers and systems to obtain meaningful information from images, videos, and other visual inputs, and take actions or recommendations based on that information. Computer vision involves giving computers the ability to discover, observe, and understand.

[0089] Related technologies often involve a variety of image-related services such as image or video classification, multimodal classification, image or video retrieval, and multimodal retrieval. Among them, multimodality refers to multiple modes, and common modes include images, voices, texts, etc. Video is a typical multimodal content, including video frames (images), soundtracks and dubbing (voice), and titles (text). For each service, in order to complete the corresponding service, it is necessary to extract features from the image or video through the image feature extraction model of the corresponding service. Therefore, for the same content, if different services are involved, it is necessary to perform multiple calculations on the same content, which consumes a lot of computing resources.

[0090] For example, for a video frame p from a video, it is necessary to extract image features from the video frame p through an image extraction model to obtain image features suitable for the service, and then perform video frame classification services, multimodal classification services, image retrieval services, video frame retrieval services, and multimodal retrieval services according to the corresponding image features. For the video frame classification service, such as Figure 1A As shown in FIG. 1 , it is a flow chart of the image or video classification service. The video frame p is input into the image feature extraction model 1 to obtain the image feature 1, which is the feature required for image classification. The image feature 1 is input into the classification layer to obtain the classification and labeling results. The classification layer is the service model of the image feature extraction model. For multimodal classification services, such as Figure 1B As shown in the figure, it is a flowchart of the multimodal classification business. The video frame p is input into the image feature extraction model 2 to obtain the image feature 2, which is the image information required for multimodal semantic understanding. The text information is input into the text feature extraction model to obtain the text feature 1. The image feature 2 and the text feature 1 are input into the multimodal feature extraction model to obtain the multimodal feature, and then the classification and labeling results are obtained through the classification layer. For the video frame retrieval business, such as Figure 1C As shown in FIG. 1 , it is a flowchart of the video frame retrieval service. The video frame p is input into the image feature extraction model 3 to obtain the image feature 3. Then, the image extraction model 3 performs feature extraction on multiple base video frames to obtain the base image feature 1 corresponding to each base video frame. The similarity between the image feature 3 and each base image feature 1 is determined, and the label or classification corresponding to the base image feature 1 with the higher similarity is used as the result of the video frame. For the multimodal retrieval service, such as Figure 1DAs shown, it is a flowchart of the multimodal retrieval business. The video frame p is input into the image feature extraction model 4 to obtain the image feature 4, and the text information is input into the text feature extraction model 2 to obtain the text feature 2. Correspondingly, the base library image feature 2 is obtained according to the base library video frame and the image feature extraction model 4, and the base library text feature is obtained according to the base library text information and the text feature extraction model 2. In addition to the same-modal similarity comparison (image to image, text to text), cross-modal comparison (image to text, text to image) is also performed.

[0091] It can be seen that for the same video frame, when its image features are applied to different businesses, the image feature extraction model is required to extract the video frame. This results in the same content features being processed multiple times by different models when applied to different businesses, and each processing generates a lot of computation. Therefore, when multiple businesses are involved, the image feature extraction process will consume a lot of computing resources.

[0092] The image feature acquisition method provided in the embodiment of the present application has the following characteristics: since the general image feature extraction model is obtained by jointly training the image feature extraction model and the text feature extraction model through multiple training samples, during the training process, the text information of the sample image is used as the supervision information, and the supervision information of the image in the specific business scenario is not used, so that the trained image feature extraction model has the ability to mine the common information of the target images suitable for different business parties. Therefore, when a feature query request carrying the target image identifier sent by the business party is received, the general image feature of the target image obtained by calling the general image feature extraction model to extract the target image corresponding to the target image identifier is returned, and the target image identifier and the general image feature of the target image are stored in the general image feature library in correspondence. In this way, when a feature query request carrying the target image identifier sent by any business party is received again, the general image feature corresponding to the target image identifier is directly returned, thereby avoiding repeated calculations of the same image by different business parties and reducing the computing consumption caused by repeated calculations.

[0093] For example, Figure 2 The figure is a flow chart of the image feature acquisition method of the embodiment of the present application. The video frame p is input into the universal image feature extraction model to obtain the universal image feature corresponding to the video frame p. The universal image feature is directly used in various video frame classification services, various multimodal classification services, various video frame retrieval services, and various multimodal retrieval services. That is, compared with the above Figures 1A to 1D According to the solution, the embodiment of the present application only requires one image feature extraction process for the same content, which greatly optimizes the process of obtaining common information of video frames and greatly reduces the consumption of computing resources.

[0094] The image feature acquisition method provided in the embodiment of the present application can be applied to Figure 3 In the application environment shown. Among them, the terminal 302 communicates with the server 304 through the network. The data storage system can store the data that the server 304 needs to process. For example, the data storage system can be a general image feature library or a directly accessible image library that the server 304 can directly access. The data storage system can be a separate device, or it can be integrated on the server 304, or integrated on the cloud or other servers. The terminal 302 and the server 304 can be used alone to execute the image feature extraction method provided in the embodiment of the present application. The terminal 302 and the server 304 can also be used in conjunction to execute the image feature extraction method provided in the embodiment of the present application.

[0095] The present application embodiment mainly uses the server alone to execute the image feature extraction method provided in the present application embodiment as an example for explanation. In one embodiment, the image feature acquisition method provided in the present application embodiment can be executed by the server 304. When the server 304 receives a feature query request carrying a target image identifier sent by a business party (for example, the terminal 302 or another server), the server 304 returns the general image features of the target image obtained by calling the general image feature extraction model to extract features from the target image corresponding to the target image identifier; wherein the general image feature extraction model is obtained by model-joint training of the image feature extraction model and the text feature extraction model, each training sample includes a sample image and text information of the sample image, and the image feature extraction model is updated according to a matching loss function determined based on multiple training samples, and the matching loss function is determined according to the degree of matching between the image features obtained by extracting features from the sample image through the image feature extraction model and the text features obtained by extracting features from the text information through the text feature extraction model; the server 304 stores the target image identifier and the general image features of the target image in the general image feature library in correspondence, so as to directly return the general image features corresponding to the target image identifier when receiving a feature query request carrying the target image identifier sent by any business party again.

[0096] The terminal 302 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 304 may be implemented by an independent server or a server cluster consisting of multiple servers, or by a cloud server.

[0097] In one embodiment, Figure 4As shown, a method for acquiring image features is provided, which can be executed by a computer device, and the computer device can be a terminal or a server. In this embodiment, the method is described by taking the method executed by a server as an example, and includes the following steps:

[0098] Step 402, when receiving a feature query request carrying a target image identifier sent by the business party, return to call the general image feature extraction model to extract the general image features of the target image corresponding to the target image identifier; wherein the general image feature extraction model is obtained by jointly training the image feature extraction model and the text feature extraction model, each training sample includes a sample image and text information of the sample image, the image feature extraction model is updated according to a matching loss function determined based on multiple training samples, and the matching loss function is determined according to the matching degree between the image features obtained by extracting features of the sample image through the image feature extraction model and the text features obtained by extracting features of the text information through the text feature extraction model.

[0099] Among them, the business party refers to any party that needs to use the universal image features. The business party may refer to any computer device that needs to use the universal image features, for example, a server or a terminal, on which a business that needs to use the universal image features is deployed. The business may be, for example, an image classification business, an image detection business, etc. The computer device may be the same computer device as the computer device that executes the image feature acquisition method provided in the embodiment of the present application, or it may be a different computer device. It can be understood that the essence of the business is a computer program that implements the corresponding business functions. Optionally, the business involving the use of universal image features may also include visual question answering, visual reasoning, visual joint reasoning, image recognition, multimodal classification, multimodal recognition, image retrieval, multimodal retrieval, and the like.

[0100] It can be understood that each business can be applied to each link of multimedia content processing. The multimedia content can be an image or a multimodal content including an image. Exemplarily, for any multimedia content, in the link of security review, it is necessary to identify the multimedia content, such as identifying whether there is sensitive content in the image. In the standardized review, it is necessary to classify the multimedia content, such as labeling the image and classifying the content. In the retrieval link, it is necessary to retrieve the multimedia content, such as deduplicating the image content, that is, to determine whether the content of the image has been processed by searching the image. In the personalized recommendation link, the semantic vector of the multimedia content is calculated to achieve accurate recommendation of multimedia content.

[0101] The target image is the image to be processed. The target image can be a video frame from a video, or it can be taken by an image acquisition device, or it can be obtained from a sharing application client, without specific limitation. The image identifier is used to distinguish each target image. Each target image has a unique corresponding image identifier. The image identifier can be an ID (Identity document), MD5 (Message-Digest Algorithm), etc. Optionally, if the target image is a video frame, the target image identifier can also represent the video to which the video frame belongs, that is, the target image identifier of the video frame represents the frame number of the video to which the video frame belongs.

[0102] As mentioned above, for the same image, in different business scenarios, it may be subjected to feature extraction by the image feature extraction model of the corresponding business scenario to obtain image features suitable for the corresponding business scenario, which are then processed by the business computer program of the corresponding business scenario. In order to avoid the same image being extracted multiple times by image feature extraction models of different business scenarios, the universal image feature extraction model of the present application can be used to extract universal image features applicable to various businesses, that is, for the same image in different businesses, there is no need to repeatedly extract image features, and only the universal image features need to be used directly. The universal image features represent the common information required by each business.

[0103] In view of this, a general image feature extraction model for extracting general image features is constructed based on deep learning. The deep learning is an artificial intelligence algorithm based on deep convolutional neural network, which is a data-driven algorithm. Unlike artificial rules, the neural network automatically mines deep Robust features by learning massive data, thus forming an automatic calculation process from input to output without human participation.

[0104] Based on the various businesses mentioned above, all businesses involve image processing, and the image processing in each business can be attributed to image classification processing. For example, for the image retrieval business, image retrieval can be converted into image classification for processing. For example, for the multimodal classification business, it involves the classification of image modalities. Therefore, in order to ensure the accuracy of the general image feature extraction model, it can be trained through the idea of ​​image classification.

[0105] On this basis, in order to further improve the accuracy of the general image feature extraction model, the image classification should be as detailed and general as possible. At this time, in the process of model training, the correlation between training text and image can be used to ensure that the classification is infinitely detailed, that is, the text can describe the image well. Therefore, the matching loss function of the model is constructed by the matching degree between the image features of the sample image and the text features of the text information, so as to obtain a general image feature extraction model with higher classification accuracy, and also obtain general image features applicable to multiple business scenarios.

[0106] For example, Figure 5 As shown, it is a schematic diagram of the process of training a general image feature extraction model in an embodiment. For each sample image, the sample image is input into the image feature extraction model to be trained to obtain image features, and the text information corresponding to the sample image is input into the text feature extraction model to be trained to obtain text features. Then, the matching loss function is determined according to the matching degree between the image features and the text features, and the trained image feature extraction model, i.e., the general image feature extraction model, is obtained according to the matching loss function.

[0107] It should be noted that the training process or update process of the universal image feature extraction model can be performed by a computer device that executes the image feature acquisition method, that is, the computer device that executes the image feature acquisition method can execute the training steps of the universal image feature extraction model to obtain the universal image feature extraction model or an updated universal image feature extraction model. Of course, the training steps of the universal image feature extraction model can also be executed by other computer devices (referred to as model providers) to obtain a universal feature extraction model or an updated universal image feature extraction model. The computer device that executes the image feature acquisition method provided in the embodiment of the present application can obtain the universal feature extraction model or the updated universal image feature extraction model from the model provider.

[0108] Optionally, the computer device obtains a general image feature extraction model, which is obtained through the above steps, that is, training based on a sample image of a training sample and text information of the sample image. After the computer device receives a feature query request carrying a target image identifier sent by the business party, the general image feature extraction model is called to perform feature extraction on the target image corresponding to the target image identifier to obtain the general image features of the target image, and in response to the feature query request, the general image features of the target image are returned.

[0109] Exemplarily, a computer device obtains a general image feature extraction model, which is obtained by the above steps, that is, training based on the sample image of the training sample and the sample image. The computer device receives a feature query request carrying a target image identifier sent by a business party. If the computer device recognizes the target image identifier for the first time, the computer device determines that the feature query request carrying the target image identifier is received for the first time, calls the general image feature extraction model to extract features of the target image corresponding to the target image identifier, obtains the general image features of the target image, and returns the general image features of the target image in response to the feature query request. If the computer device does not recognize the target image identifier for the first time, the computer device determines that the feature query request carrying the target image identifier is not received for the first time. At this time, the computer device returns an error code to the business party to indicate that the business party initiates a feature acquisition request. At this time, the computer device receives the feature acquisition request carrying the target image identifier sent by the business party, and directly returns the general image features corresponding to the target image identifier according to the feature acquisition request. The return of the error code is used to remind the business party that the general image features corresponding to the target image identifier do not need to be calculated repeatedly, that is, the general image features have been stored in the general image feature library.

[0110] Step 404, the target image identifier and the universal image feature of the target image are stored in a universal image feature library so that when a feature query request carrying the target image identifier is received again from any business party, the universal image feature corresponding to the target image identifier is directly returned.

[0111] Among them, the universal image feature library is used to store the universal image features corresponding to the image. In one embodiment, in order to facilitate the query of features and reduce the query delay, the computer device can set multiple universal image feature libraries, each of which corresponds to each business party, that is, each universal image feature library stores a universal image feature requested by a business party. For example, for the image classification business, there is a corresponding universal image feature library 1, which stores the universal image features corresponding to all images under the business. For another example, the universal image feature library stores the universal image features requested by all business parties. The universal image feature library can be a storage unit in the computer device, or it can be a storage unit in the computer device. In this case, the computer device directly accesses the universal image feature library to obtain or store the universal image features corresponding to the target image identifier. It can be understood that the universal image feature library is a storage space accessible to the computer device. For example, if the computer device is a server, the universal image feature library can be a database of the server. If the computer device is a terminal, the universal image feature can be another computer device with storage function that can be directly accessed by the terminal.

[0112] Optionally, the computer device stores the target image identifier and the universal image feature of the target image in a universal image feature library, and updates the mapping relationship between the image identifier and the universal image feature based on the target image identifier and the universal image feature corresponding thereto, so that when a feature query request carrying the target image identifier is received again from any business party, the computer device directly returns the universal image feature corresponding to the target image identifier according to the updated mapping relationship.

[0113] Exemplarily, the computer device stores the target image identifier and the universal image feature of the target image in a universal image feature library, and based on the target image identifier and the universal image feature corresponding thereto, updates the mapping relationship between the image identifier and the universal image feature to obtain an updated mapping relationship. When the computer device receives a feature query request carrying the target image identifier from any business party again, the updated mapping relationship is used as the mapping relationship corresponding to the feature query request carrying the target image identifier received again. The computer device determines whether there is a universal image feature corresponding to the target image identifier based on the mapping relationship corresponding to the feature query request carrying the target image identifier received again. If there is, that is, the universal image identifier corresponding to the target image identifier is stored, the computer device directly returns the universal image feature corresponding to the target image identifier.

[0114] like Figure 6 As shown, it is a schematic diagram of applying universal image features in an embodiment. The business party of image classification sends a feature query request to the computer device, and the computer device verifies that the feature query request is received for the first time. Then, after obtaining the universal image feature corresponding to the target image according to the universal image feature extraction model, for the business party of image classification, the business party processes the universal image feature through the classification layer 1 to obtain the image classification result. For the business party of multimodal classification, after obtaining the universal image feature, the business party also obtains the text feature 1, and performs multimodal cross calculation based on the universal image feature and the text feature 1 to obtain the multimodal cross calculation result. The business party processes the multimodal cross calculation result through the classification layer 2 to obtain the multimodal classification result. For the business party of image retrieval, the business party directly compares the similarity between the universal image feature and the base database feature to obtain the image retrieval result. For the business party of multimodal retrieval, after obtaining the universal image feature, the business party also obtains the text feature 2, and obtains the multimodal retrieval result based on the universal image feature, the text feature 2, the base database image feature and the base database text feature. It can be seen that the present application only needs one image feature extraction to obtain universal image features applicable to various businesses, which greatly reduces the consumption of computing resources.

[0115] In the above-mentioned image feature acquisition method, since the general image feature extraction model is obtained by jointly training the image feature extraction model and the text feature extraction model, during the training process, the text information of the sample image is used as the supervisory information, and the supervisory information of the image in the specific business scenario is not used, so that the trained image feature extraction model has the ability to mine the common information of the target images suitable for different business parties. Therefore, when a feature query request carrying the target image identifier sent by the business party is received, the general image feature of the target image obtained by calling the general image feature extraction model to extract the target image corresponding to the target image identifier is returned, and the target image identifier and the general image feature of the target image are stored in the general image feature library in correspondence. In this way, when a feature query request carrying the target image identifier sent by any business party is received again, the general image feature corresponding to the target image identifier is directly returned, avoiding repeated calculations of the same image by different business parties, and reducing the computing consumption caused by repeated calculations.

[0116] In one embodiment, upon receiving a feature query request carrying a target image identifier sent by a business party, returning a general image feature of the target image obtained by calling a general image feature extraction model to perform feature extraction on a target image corresponding to the target image identifier, including: upon receiving a feature query request carrying a target image identifier sent by a business party, obtaining a target image corresponding to the target image identifier according to the target image identifier, calling a general image feature extraction model to perform feature extraction on the target image to obtain a general image feature corresponding to the target image identifier, and returning the general image feature corresponding to the target image identifier in response to the feature query request.

[0117] Optionally, when the computer device receives a feature query request carrying a target image identifier sent by the business party, the computer device may query a target image corresponding to the target image identifier from the image library according to the target image identifier. The computer device calls the universal image feature extraction model to extract features from the target image obtained by the query, obtains universal image features corresponding to the target image identifier, and returns the universal image features corresponding to the target image identifier in response to the feature query request.

[0118] The image library is used to store images. The image library is a storage space for storing images that can be directly accessed by a computer device that executes the image feature acquisition method provided in the embodiment of the present application. The image library also stores the corresponding relationship between the image identifier and the image. For example, if the computer device is a server, the image library can be a database of the server. If the computer device is a terminal, the image library can be another computer device that provides an image storage function and can be directly accessed by the terminal.

[0119] Exemplarily, when the computer device receives a feature query request carrying a target image identifier sent by a business party for the first time, the computer device queries the target image from the image library according to the target image identifier and obtains the target image corresponding to the target image identifier. The computer device automatically calls the universal image feature extraction model to extract features from the target image, obtains universal image features corresponding to the target image identifier, and stores the universal image features by itself, and returns the universal image features corresponding to the target image identifier in response to the feature query request.

[0120] Alternatively, when the computer device first receives a feature query request carrying a target image identifier sent by a business party, the computer device returns information to the business party that there is no universal image feature corresponding to the target image identifier, so as to remind the business party. The computer device receives a feature extraction request carrying a target image identifier sent by the business party, and the computer device queries the target image from the image library and obtains the target image corresponding to the target image identifier based on the target image identifier in the feature extraction request. The computer device calls the universal image feature extraction model to extract features from the target image and obtains the universal image features corresponding to the target image identifier. The computer device stores the universal image features corresponding to the target image identifier and returns the universal image features corresponding to the target image identifier to the business party.

[0121] For example, after the computer device stores the universal image feature corresponding to the target image identifier, the computer device initiates a feature acquisition request. If the computer device cannot obtain the universal image feature at this time, it indicates that the storage is abnormal. At this time, the computer device returns an abnormal prompt message to the business party. In other words, the feature extraction request and the feature acquisition request are two independent requests. The feature extraction request is used to initiate the extraction process of the universal image feature. After storage, it is still necessary to receive the feature acquisition request from the business party to instruct the computer device to obtain it from the stored universal image feature library. If it cannot be obtained, it indicates that the storage operation is incorrect.

[0122] It should be noted that in the related technologies, for the same image or video, each business party will involve transmitting the image or video and obtaining the image or video when performing corresponding feature reasoning. At this time, data leakage and copyright leakage are very likely to occur during the transmission or acquisition process.

[0123] In this application, any business party only needs to send a relevant request carrying the target image identifier to the computer device, so that the computer device can obtain the target image from the independent image library. In other words, during the entire business processing process, the business party is not involved in the transmission of any image or video, nor is it involved in the acquisition of any image or video, which greatly avoids the risk of data leakage and copyright leakage.

[0124] In this embodiment, when a feature query request carrying a target image identifier is received from a business party, a target image corresponding to the target image identifier is obtained according to the target image identifier, thereby avoiding data leakage and copyright leakage caused by the business party directly obtaining the target image, and ensuring the security of each business processing process. After obtaining the target image, the general image feature extraction model can be called to timely and accurately extract the general image features applicable to each business, thereby improving the efficiency of the business in the general image feature acquisition process.

[0125] In one embodiment, when a feature query request carrying a target image identifier is received again from any business party, the universal image feature corresponding to the target image identifier is directly returned, including: when a feature query request carrying a target image identifier is received again from any business party, based on the mapping relationship between the image identifier and the universal image feature, it is queried whether there is a universal image feature corresponding to the target image identifier in the universal image feature library. If it is found that there is a universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, the universal image feature corresponding to the target image identifier queried from the universal image feature library is returned. If it is found that there is no universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, an abnormal prompt message is returned.

[0126] Optionally, when the computer device receives a feature query request carrying the target image identifier sent by any business party again, the computer device uses the updated mapping relationship as the mapping relationship corresponding to the feature query request carrying the target image identifier received again. The computer device queries whether there is a universal image feature corresponding to the target image identifier in the universal image feature library based on the mapping relationship corresponding to the feature query request carrying the target image identifier received again. If it is found that there is a universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, an error code is returned to the business party to indicate that the business party initiates a feature acquisition request, and the computer device receives the feature acquisition request carrying the target image identifier sent by the business party, and according to the feature acquisition request, directly returns the universal image feature corresponding to the target image identifier obtained from the universal image feature library. If it is found that there is no universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, an abnormal prompt message is returned.

[0127] Exemplarily, if it is found that there is no universal image feature corresponding to the target image identifier in the universal image feature library, the computer device determines that the universal image feature storage is abnormal, and returns abnormal prompt information in response to the feature query request.

[0128] In this embodiment, when a feature query request carrying the target image identifier is received again from any business party, based on the mapping relationship between the image identifier and the universal image feature, it is possible to accurately query whether the universal image feature corresponding to the target image identifier is successfully stored in the universal image feature library, that is, to determine whether the universal image feature exists. If it exists, the universal image feature required by any business party can be quickly and accurately returned; if it does not exist, it can be timely reflected that the universal image feature library has failed to cache the universal image feature corresponding to the target image identifier.

[0129] In one embodiment, the method further includes: when an updated universal image feature extraction model is received, clearing the universal image features stored in the universal image feature library.

[0130] Optionally, when the computer device receives an updated universal image feature extraction model, or when the computer device retrains the model based on new training samples to obtain an updated universal image feature extraction model, the computer device clears the universal image features stored in the universal image feature library.

[0131] It should be noted that the universal image features cleared by the computer device are obtained by updating the universal image feature extraction model before. To ensure the accuracy of each business, once the universal image feature extraction model is updated, it is not necessary to retain the universal image features obtained by the universal image feature extraction model before the update. In addition, by timely clearing the universal image features obtained according to the general image feature extraction model that has not been updated, storage space can be saved and costs can be reduced.

[0132] In this embodiment, once an updated general image feature extraction model is received, the stored general image features are directly cleared to ensure the accuracy of subsequent business processing processes. At the same time, storage space can be saved and costs can be reduced.

[0133] In one embodiment, the method further includes: periodically clearing the universal image feature library. For example, the computer device may periodically clear the universal image feature library, that is, periodically delete all universal image features cached in the universal image feature library. For another example, the computer device may periodically delete some universal image features, and clear the universal image features after the universal image features have been cached in the universal image feature library for a preset time period.

[0134] In this embodiment, after the universal image features are stored for a preset period of time, the universal image features stored in the universal image feature library are cleared in time, which can save storage space and reduce costs.

[0135] In one embodiment, the method also includes: if a feature query request carrying a target image identifier is received from a business party, returning an updated universal image feature of the target image obtained by calling an updated universal image feature extraction model to extract features from the target image corresponding to the target image identifier; storing the target image identifier and the updated universal image feature of the target image in a universal image feature library in correspondence with each other, so that when a feature query request carrying the target image identifier is received from any business party again, the updated universal image feature corresponding to the target image identifier is directly returned.

[0136] Optionally, after the computer device clears the universal image features stored in the universal image feature library, after the computer device receives a feature query request carrying the target image identifier sent by the business party, if the computer device verifies that the feature query request carrying the target image feature identifier is received after the model is updated, the updated universal image feature extraction model is called to perform feature extraction on the target image corresponding to the target image identifier, obtain the updated universal image features of the target image, and return the updated universal image features of the target image.

[0137] The computer device stores the target image identifier and the updated universal image feature of the target image in a universal image feature library in correspondence, and based on the target image identifier and the corresponding updated universal image feature, updates the mapping relationship between the image identifier and the updated universal image feature to obtain an updated mapping relationship, so that when a feature query request carrying the target image identifier is received again from any business party, the computer device directly returns the updated universal image feature corresponding to the target image identifier according to the updated mapping relationship.

[0138] Exemplarily, after the computer device clears the general image features stored in the general image feature library, the computer device receives a feature query request carrying a target image identifier sent by a business party. If the computer device recognizes the target image identifier for the first time after the model is updated, the computer device determines that this is the first time the feature query request carrying the target image identifier has been received after the model is updated. Then, the computer device calls the updated general image feature extraction model to perform feature extraction on the target image corresponding to the target image identifier, obtains the updated general image features of the target image, and returns them to the business party.

[0139] The computer device stores the target image identifier and the updated universal image feature of the target image in a universal image feature library, and based on the target image identifier and the corresponding updated universal image feature, updates the mapping relationship between the image identifier and the updated universal image feature. When the computer device receives a feature query request carrying the target image identifier from any business party again, the updated mapping relationship corresponding to the feature query request received for the first time is used as the mapping relationship corresponding to the feature query request carrying the target image identifier received again. The computer device determines whether there is an updated universal image feature corresponding to the target image identifier based on the mapping relationship corresponding to the feature query request received again carrying the target image identifier. If there is, that is, the universal image identifier corresponding to the target image identifier is stored, the computer device directly returns the updated universal image feature corresponding to the target image identifier.

[0140] In this embodiment, after the general image feature extraction model is updated, once the general image features of the target image need to be calculated, the updated general image features suitable for each business can be obtained again based on the updated general image feature extraction model, so as to ensure that each subsequent business can perform effective inference calculations based on the accurate updated general image features.

[0141] In one embodiment, before the computer device calls the universal image feature extraction model, the universal image feature extraction model is obtained through model training. As mentioned above, the training process or update process of the universal image feature extraction model can be performed by a computer device that executes the image feature acquisition method provided by the embodiment of the present application, that is, before calling the universal image feature extraction model, the image feature acquisition method provided by the embodiment of the present application may also include the step of training the universal image feature extraction model. The training process or update process of the universal image feature extraction model may also be performed by other computer devices (referred to as model providers) to obtain a universal feature extraction model or an updated universal image feature extraction model. The computer device that executes the image feature acquisition method provided by the embodiment of the present application may obtain the universal feature extraction model or an updated universal image feature extraction model from the model provider.

[0142] In one embodiment, Figure 7 As shown, the training steps of the general image feature extraction model include:

[0143] Step 702: Acquire multiple training samples, each training sample including a sample image and text information corresponding to the sample image.

[0144] Each training sample corresponds to a video, so for each training sample, the sample image in the training sample can be the cover image of the video, or a video frame randomly sampled from the video, and correspondingly, the text information in the training sample is the title of the video, the narration in the image, etc. The video can be a short video, a long video, etc.

[0145] Optionally, when training a general image feature extraction model, the computer device may obtain a plurality of sample tuples, which may be a combination of an image and text information, or a combination of a video and text information. When the sample tuple is a combination of an image and text information, the image of each sample tuple may be a cover image of a video, and the text information may be a title of the video. At this time, the image and text information in each sample tuple are used as the training image and text information of the training image in each training sample, respectively. When the sample tuple is a combination of a video and text, for each video in the sample tuple, the video is randomly sampled to obtain a video frame, and the title of the video is used as the text information. At this time, the video frame and text information in each sample tuple are used as the training image and text information of the training image in each training sample, respectively.

[0146] Step 704: extract features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample.

[0147] Among them, the computer device can build an initial image feature extraction model, which is an untrained image feature extraction model, and is a model that is to learn the ability to extract image features through training samples. The image feature extraction model can be built based on a neural network, such as a convolutional neural network, a recursive neural network, etc. The embodiment of the present application does not limit the model structure of the image feature extraction model.

[0148] Optionally, for each training sample, the computer device inputs the sample image in the training sample into the image feature extraction model, and performs feature extraction on the sample image of the training sample through the image feature extraction model to obtain the image feature of the training sample.

[0149] Step 706: extract features from the text information of each training sample using a text feature extraction model to obtain text features of each training sample.

[0150] The computer device can build an initial text feature extraction model, which is an untrained text feature extraction model, and is a model that needs to learn the ability to extract text features through training samples. The text feature extraction model can be built based on a neural network, such as a convolutional neural network, a recursive neural network, etc. The embodiment of the present application does not limit the model structure of the text feature extraction model.

[0151] Optionally, for each training sample, the computer device inputs the text information corresponding to the sample image in the training sample into the text feature extraction model, and performs feature extraction on the text information of the training sample through the text feature extraction model to obtain the text feature of the training sample.

[0152] It can be understood that the text feature extraction model is a model that assists the image feature extraction model in the present application in training. The computer device jointly trains the image feature extraction model and the text feature extraction model, so that the two can supervise each other's learning during the training process.

[0153] Step 708: construct a positive example matching loss based on the image features and text features of the same training sample.

[0154] Among them, the positive example matching loss represents the degree of difference between the image features and text features belonging to the same training sample.

[0155] Step 710: construct a negative example matching loss based on the image features and text features of each pair of training samples.

[0156] Among them, the negative example matching loss represents the degree of difference between the image features and text features belonging to different training samples.

[0157] Step 712: Determine a matching loss function according to the positive example matching loss and the negative example matching loss.

[0158] Optionally, the computer device obtains a matching loss function according to the positive example matching loss and the negative example matching loss. The matching loss function is positively correlated with the positive example matching loss and positively correlated with the negative example matching loss.

[0159] Step 714, jointly train the image feature extraction model and the text feature extraction model according to the matching loss function, and use the trained image feature extraction model as a general image feature extraction model.

[0160] Optionally, the computer device iteratively jointly trains the image feature extraction model and the text feature extraction model according to the matching loss function until the training stop condition is reached, thereby obtaining a trained image feature extraction model, and using the trained image feature extraction model as a general image feature extraction model.

[0161] Exemplarily, the computer device iteratively jointly trains the image feature extraction model and the text feature extraction model according to the matching loss function until the loss value of the matching loss function of that iteration meets the loss threshold, thereby obtaining a trained image feature extraction model, and using the trained image feature extraction model as a general image feature extraction model.

[0162] Alternatively, the computer device iteratively jointly trains the image feature extraction model and the text feature extraction model according to the matching loss function until the number of iterations meets the preset number of iterations, thereby obtaining a trained image feature extraction model, and using the trained image feature extraction model as a general image feature extraction model.

[0163] In one embodiment, referring to Figure 7 After obtaining the general image feature extraction model, the general image feature extraction model can be used by any business party. Figure 7 , and may also include:

[0164] Step 716a, calling the universal image feature extraction model to obtain the universal image features of the image, performing image classification based on the universal image features, and obtaining an image classification result.

[0165] For example, a computer device that deploys an image classification service may call the general image feature extraction model to obtain the general image features of the image.

[0166] Step 716b, calling the universal image feature extraction model to obtain the universal image features of the image, performing image detection based on the universal image features, and obtaining image detection results.

[0167] For example, a computer device that deploys an image detection service can call the general image feature extraction model to obtain the general image features of the image and perform image detection.

[0168] Step 716c, calling the universal image feature extraction model to obtain the universal image features of the image, performing image recognition based on the universal image features, and obtaining an image recognition result.

[0169] For example, a computer device that deploys an image recognition service can call the general image feature extraction model to obtain the general image features of the image and perform image recognition.

[0170] In this embodiment, a plurality of training samples are obtained, each of which includes a sample image and text information corresponding to the sample image. The image feature extraction model is used to extract features of the sample image of each training sample to obtain the image features of each training sample. The text feature extraction model is used to extract features of the text information of each training sample to obtain the text features of each training sample. In this way, each training sample corresponds to image features and text features that match it. A positive example matching loss is constructed based on the image features and text features of the same training sample to reflect the gap between the matching image features and text features. A negative example matching loss is constructed based on the image features and text features of each pair of training samples to reflect the difference between the mismatched image features and text features. A matching loss function is determined based on the positive example matching loss and the negative example matching loss. According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained. During the training process, the text information of the sample image is used as the supervisory information, and the supervisory information of the image in the specific business scenario is not used. In this way, the trained image feature extraction model has the ability to mine the common information of target images suitable for different business parties. In this way, when the trained image feature extraction model is used as a general image feature extraction model, it can accurately extract general image features suitable for any business.

[0171] In one embodiment, referring to Figure 7 , step 708, constructing a positive example matching loss based on the image features and text features of the same training sample, including:

[0172] Step 708a, for each training sample, calculate the similarity between the image feature and the text feature of the training sample to obtain the intra-sample feature matching degree of each training sample;

[0173] Step 708b: Determine the positive example matching loss according to the intra-sample feature matching degree of each training sample.

[0174] Optionally, for each training sample, the computer device calculates the similarity between the image feature and the text feature of the training sample to obtain the in-sample feature matching degree of the training sample. The cross entropy of the in-sample feature matching degree of each training sample is calculated, and the cross entropy of the in-sample feature matching degree of each sample is fused to determine the positive example matching loss. The in-sample feature matching degree is inversely correlated with the positive example matching loss.

[0175] Exemplarily, for each training sample, the computer device calculates the similarity between the image features and the text features of the training sample, and uses the similarity as the in-sample feature matching degree of the training sample. For each in-sample feature matching degree, the negative of the logarithm of the in-sample feature matching degree with a preset exponent as the base is used as the cross entropy of the in-sample feature matching degree. The cross entropies of the in-sample feature matching degrees are fused to determine the positive example matching loss. The in-sample feature matching degree is inversely correlated with the positive example matching loss. Among them, the similarity can be calculated by cosine similarity distance calculation. Among them, the cosine similarity calculation: the cosine value of the angle between two vectors in the vector space is used as a measure of the size of the difference between two individuals. The larger the cosine distance, the higher the similarity between the two vectors. In addition, the calculation of the similarity can also be determined by Euclidean distance and Pearson correlation coefficient, which are not specifically limited.

[0176] For example, for training sample C1, the similarity between the image feature x1 and the text feature y1 of the training sample C1 is P1, that is, the feature matching degree within the sample is P1. At this time, the cross entropy of the feature matching degree within the sample is the negative of the logarithm of P1 with the base e as the exponent, that is, -ln(P1).

[0177] In this embodiment, for each training sample, by calculating the similarity between the image feature and the text feature of the training sample, the positive example matching loss can be obtained to accurately reflect the degree of difference between the matched sample image and text information.

[0178] In one embodiment, referring to Figure 7 , step 710, constructing a negative example matching loss based on the image features and text features of the pairwise training samples, including:

[0179] Step 710a, for each training sample, calculating the similarity between the image feature of the training sample and the text features of each training sample other than the training sample, and obtaining the inter-sample feature matching degree of each training sample;

[0180] Step 710b: determine the negative example matching loss according to the inter-sample feature matching degree of each training sample.

[0181] Optionally, for each training sample, the computer device calculates the similarity between the image feature of the training sample and the text feature of each training sample other than the training sample, and obtains the inter-sample feature matching degree of each training sample. The cross entropy of the inter-sample feature matching degree of each training sample is calculated, and the cross entropy of the inter-sample feature matching degree is fused to obtain the negative example matching loss. The inter-sample feature matching degree is positively correlated with the negative example matching loss.

[0182] Exemplarily, for each training sample, the computer device calculates the similarity between the image features of the training sample and the text features of other training samples except the training sample, and obtains the inter-sample feature matching degree of the training sample. For each inter-sample feature matching degree, the difference between the unit value and the inter-sample feature matching degree is calculated, and the computer device uses the negative number of the logarithm of the difference with a preset exponent as the base as the cross entropy of the inter-sample feature matching degree. The cross entropy of the feature matching degree between each sample is fused to obtain the negative example matching loss. Among them, the similarity can be calculated by cosine similarity distance calculation. Among them, the cosine similarity calculation: the cosine value of the angle between two vectors in the vector space is used as a measure of the size of the difference between two individuals. The larger the cosine distance, the higher the similarity between the two vectors. In addition, the calculation of the similarity can also be determined by Euclidean distance and Pearson correlation coefficient, which is not specifically limited.

[0183] For example, for training sample C1, the image feature x1 and text feature y1 of the training sample C1. For another training sample C2, the image feature x2 and text feature y2 of the training sample C2, at this time, the similarity P2 between the image feature x1 of the training sample C1 and the text feature y2 of the training sample C2 is calculated, that is, the inter-sample feature matching degree of the training sample C1 is P2. The cross entropy of the inter-sample feature matching degree P2 is -ln(1-P2).

[0184] In this embodiment, the negative example matching loss is obtained by calculating the similarity between the image features of the training sample and the text features of other training samples, so as to accurately characterize the degree of difference between the mismatched sample images and text information.

[0185] In one embodiment, a matching loss function is determined based on the positive example matching loss and the negative example matching loss, including: determining the matching loss function based on the intra-sample feature matching degree of each training sample and the inter-sample feature matching degree, the matching loss function is inversely correlated with the intra-sample feature matching degree, and positively correlated with the inter-sample feature matching degree.

[0186] Optionally, the computer device determines the matching loss function through cross entropy processing according to the intra-sample feature matching degree of each training sample and the inter-sample feature matching degree.

[0187] It should be noted that since the intra-sample feature matching degree is inversely correlated with the positive example matching loss, and the matching loss function is positively correlated with the positive example matching loss, the smaller the intra-sample feature matching degree, the larger the matching loss function, that is, the matching loss function is inversely correlated with the intra-sample feature matching degree. Since the inter-sample feature matching degree is positively correlated with the negative example matching loss, and the matching loss function is positively correlated with the negative example matching loss, the matching loss function is positively correlated with the inter-sample feature matching degree.

[0188] In this embodiment, the matching loss function constructed by combining the feature matching degree within each sample and the feature matching degree between samples can comprehensively and accurately train the image feature extraction model, thereby obtaining a universal image feature extraction model with high accuracy.

[0189] In one embodiment, a matching loss function is determined based on the intra-sample feature matching degree of each training sample and the inter-sample feature matching degree, including: calculating the sum of the cross entropies of the feature matching degrees within each sample, calculating the sum of the cross entropies of the feature matching degrees between each sample; and determining the matching loss function based on the sum of the cross entropies of the feature matching degrees within each sample and the sum of the cross entropies of the feature matching degrees between each sample.

[0190] Optionally, the computer device obtains the cross entropy of the feature matching degree within each sample and the cross entropy of the feature matching degree between each sample, and calculates the sum of the cross entropy of the feature matching degree within each sample, and calculates the sum of the cross entropy of the feature matching degree between each sample. The computer device fuses the sum of the cross entropy of the feature matching degree within each sample and the sum of the cross entropy of the feature matching degree between each sample to determine the matching loss function.

[0191] Exemplarily, there are K training samples, corresponding to K image features and K text features. At this time, the sum of the cross entropies of the feature matching within the K samples S1 is calculated, and the sum of the cross entropies of the feature matching between the K*(K-1) samples S2 is calculated. The model provider superimposes the sum of the cross entropies of the feature matching within each sample and the sum of the cross entropies of the feature matching between each sample, that is, superimposes S1 and S2, to obtain the matching loss function.

[0192] In this embodiment, the matching loss function is constructed by summing the cross entropy of the feature matching degree within each sample and the cross entropy of the feature matching degree between each sample. Thus, the matching loss function can be used to accurately evaluate whether the image feature extraction model to be trained is well trained.

[0193] In one embodiment, the image feature extraction model and the text feature extraction model are jointly trained according to the matching loss function, and the trained image feature extraction model is used as a general image feature extraction model, including: taking minimizing the matching loss function as the goal, after updating the network parameters of the image feature extraction model and the text feature extraction model, returning to the step of obtaining multiple training samples to continue training until the training stop condition is met, to obtain a general image feature extraction model.

[0194] Optionally, the computer device updates the network parameters of the image feature extraction model and the text feature extraction model respectively through gradient descent processing, and returns to the step of obtaining multiple training samples according to the updated network parameters of the image feature extraction model and the updated network parameters of the text feature extraction model to continue training until the matching loss function is minimized, thereby obtaining a universal image feature extraction model.

[0195] Exemplarily, for each iteration, multiple training samples of the iteration are obtained, and according to the aforementioned steps 702 to 706, the image features obtained by the image feature extraction model and the text features obtained by the text feature extraction model can be obtained, and the loss value is calculated by matching the loss function according to the image features and the text features. If the loss value does not meet the loss threshold, the derivatives of the loss value and the image network parameters of the image feature extraction model and the text network parameters of the text feature extraction model are calculated to obtain the image derivative corresponding to the image feature extraction model and the text derivative corresponding to the text feature extraction model. The image derivative is subtracted from the current image network parameters according to a preset ratio to obtain the updated image network parameters, and the text derivative is subtracted from the text network parameters according to a preset ratio to obtain the updated text network parameters. The updated image feature extraction model is obtained according to the updated image network parameters, and the updated text feature extraction model is obtained according to the updated text network parameters. The updated image feature extraction model and the updated text feature extraction model are used as the image feature extraction model and the text feature extraction model of the next iteration respectively, and the next iteration is entered. The step of obtaining multiple training samples of the iteration is returned to continue execution until the loss value of the iteration meets the loss threshold, and a universal image feature extraction model is obtained.

[0196] For example, Figure 8As shown, it is a flowchart of the training steps of a general image feature extraction model in an embodiment. Step 802: Construct a training data set. Step 804: For each iteration, randomly sample multiple sample tuples of the same type from the training data set. The multiple sample tuples obtained by random sampling each time can be regarded as a batch, that is, including N sample tuples, and each sample tuple determines a training sample. Step 806: Perform the forward process of the network according to each training sample to calculate the current loss value of the matching loss function corresponding to the current iteration. Specifically, according to the aforementioned steps 702 to 706, the current image features obtained by the current image feature extraction model and the current text features obtained by the current text feature extraction model can be obtained, and according to the current image features and the current text features, the current loss value is calculated through the matching loss function. Step 808: If the current loss value does not meet the loss threshold, the network parameters of all models are updated through gradient descent processing, specifically: the current loss value is calculated with the derivatives of the current image network parameters of the current image feature extraction model and the current text network parameters of the current text feature extraction model, respectively, to obtain the current image derivatives corresponding to the current image feature extraction model and the current text derivatives corresponding to the current text feature extraction model. The current image derivatives are subtracted from the current current image network parameters according to a preset ratio to obtain the updated current image network parameters, and the current text derivatives are subtracted from the current text network parameters according to a preset ratio to obtain the updated current text network parameters. Then, an updated image feature extraction model is obtained based on the updated current image network parameters, and an updated text feature extraction model is obtained based on the updated current text network parameters. The updated image feature extraction model and the updated text feature extraction model are used as the current image feature extraction model and the current text feature extraction model of the next iteration respectively, and the next iteration is entered, and the step of randomly selecting multiple sample pairs of the same type from the training data set is continued until the current loss value of the iteration meets the loss threshold, and the general image feature extraction model is obtained. Step 810: Save the network parameters of the general image feature extraction model.

[0197] In this embodiment, when the model is trained with the goal of minimizing the matching loss function, it can ensure that the image features and text features of the same training sample have a higher degree of matching, and the image features of the training sample have a lower degree of matching with the text features of other training samples. Thus, a more robust and accurate general image feature extraction model can be obtained.

[0198] In one embodiment, Fig. 9As shown, a training method for a general image feature extraction model is provided, and the method is applied to a computer device as an example for explanation, including the following steps 902 to 914. The computer device may be a terminal or a server. It should be noted that the computer device may be the same as the computer device that executes the image feature acquisition method, or may be different from the computer device that executes the image feature acquisition method.

[0199] Step 902: Acquire multiple training samples, each training sample including a sample image and text information corresponding to the sample image.

[0200] Each training sample corresponds to a video, so for each training sample, the sample image in the training sample can be the cover image of the video, or a video frame randomly sampled from the video, and correspondingly, the text information in the training sample is the title of the video, the narration in the image, etc. The video can be a short video, a long video, etc.

[0201] Optionally, the computer device obtains a plurality of sample 2-tuples, which may be a combination of an image and text information, or a combination of a video and text information. When the sample 2-tuple is a combination of an image and text information, the image of each sample 2-tuple may be a cover image of a video, and the text information may be a title of the video. At this time, the image and text information in each sample 2-tuple are used as the training image and text information of the training image in each training sample, respectively. When the sample 2-tuple is a combination of a video and text, for each video in the sample 2-tuple, the video is randomly sampled to obtain a video frame, and the title of the video is used as the text information. At this time, the video frame and text information in each sample 2-tuple are used as the training image and text information of the training image in each training sample, respectively.

[0202] Step 904: extract features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample.

[0203] Step 906: extract features from the text information of each training sample using a text feature extraction model to obtain text features of each training sample.

[0204] Step 908: construct a positive example matching loss based on the image features and text features of the same training sample.

[0205] Step 910: construct a negative example matching loss based on the image features and text features of each pair of training samples.

[0206] Step 912: Determine a matching loss function according to the positive example matching loss and the negative example matching loss.

[0207] Step 914, jointly train the image feature extraction model and the text feature extraction model according to the matching loss function, and use the trained image feature extraction model as a general image feature extraction model.

[0208] Regarding the training method of the universal image feature extraction model, reference may be made to the embodiment described in the training steps of the universal image feature extraction model in the previous text, and the description will not be repeated here. In the training method of the universal image feature extraction model, multiple training samples are obtained, and each training sample includes a sample image and text information corresponding to the sample image. Through the image feature extraction model, feature extraction is performed on the sample image of each training sample to obtain the image feature of each training sample. Through the text feature extraction model, feature extraction is performed on the text information of each training sample to obtain the text feature of each training sample. In this way, each training sample corresponds to an image feature and a text feature that matches it. A positive example matching loss is constructed based on the image features and text features of the same training sample to reflect the gap between the matching image features and text features. A negative example matching loss is constructed based on the image features and text features of the two training samples to reflect the difference between the unmatched image features and text features. A matching loss function is determined based on the positive example matching loss and the negative example matching loss. According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained. During the training process, the text information of the sample image is used as the supervisory information, and the supervisory information of the image in the specific business scenario is not used. In this way, the trained image feature extraction model has the ability to mine the common information of target images suitable for different business parties. In this way, when the trained image feature extraction model is used as a general image feature extraction model, it can accurately extract general image features suitable for any business.

[0209] The present application also provides an application scenario, which applies the above-mentioned image feature acquisition method. Specifically, the application of the image feature acquisition method in the application scenario is as follows: in various types of video clients, for videos that have not yet been uploaded or are to be uploaded, in order to facilitate the reasoning of different services for each video frame in the video, such as security review of the video frame, classification of the content of the video frame, retrieval of the video frame, etc. The image feature acquisition method of the present application can be used to obtain the general image features corresponding to each video frame. In this way, different business reasoning is performed directly based on the general image features corresponding to the video frame, reducing computational loss.

[0210] The following is a detailed application scenario: the video server receives the video uploaded by each terminal through the video client, extracts the image of the video, generates the corresponding image identifier, and stores the image and the image identifier in the image library. Then, each business party such as the video classification server, the video detection server, and the video review server successively obtains the image identifier of the image that needs to be processed from the image library (each business party can record the processed image identifier and obtain the unprocessed image identifier from the image library for processing), and initiates an image feature acquisition request to the feature extraction server based on the image identifier. When the feature extraction server receives an image feature acquisition request for any image identifier from a business party, it obtains the corresponding image from the image library according to the image identifier, calls the general image feature extraction model to extract the corresponding general image feature, returns it to the business party, and caches the image identifier and the general image feature in the general image feature library. Thereafter, when the feature extraction server receives an image feature acquisition request for the image identifier from a certain (other) business party again, it can query the cached general image features from the general image feature library according to the image identifier and return it. In this way, the business party does not need to contact the image itself, which greatly protects the image content and avoids the risk of data leakage. Moreover, it is not necessary for each business party to extract features from the same image, which avoids redundant image feature extraction processes and saves computing resources. Of course, it is not limited to this. The image feature extraction method provided by the present application can also be applied to other application scenarios. For example, for personalized recommendation scenarios of video clients, social application clients, and news clients, when personalized images or videos are recommended to the user based on the images or videos that the user has browsed in the past, the image feature acquisition method provided in the embodiment of the present application can be used to quickly and accurately obtain the general image features corresponding to the images or videos that have been browsed in the past, and perform subsequent semantic vector calculations based on the general image features to achieve precise image or video recommendations.

[0211] The above application scenarios are merely illustrative illustrations. It should be understood that the application of the image feature acquisition methods provided in the embodiments of the present application is not limited to the above scenarios.

[0212] In a specific embodiment, Fig.10As shown, it is a schematic diagram of the image feature acquisition method in this embodiment, which involves a model provider that provides a universal image feature extraction model, a business party, and a universal image feature system. The system includes a computer device that provides universal image features and a universal image feature library. The computer device includes a universal image feature extraction model obtained from the model provider, and the computer device can also read and cache features of the universal image feature library. The model provider communicates with the computer device that provides universal image features in the universal image feature system through a network, and the business party communicates with the computer device through a network. Specifically, the steps of the image feature acquisition method are as follows:

[0213] Step 1: The computer device as the model provider obtains multiple training samples, each training sample includes a sample image and text information corresponding to the sample image. Through the image feature extraction model, the sample image of each training sample is feature extracted to obtain the image feature of each training sample. Through the text feature extraction model, the text information of each training sample is feature extracted to obtain the text feature of each training sample. For each training sample, the model provider calculates the similarity between the image feature and the text feature of the training sample, and uses the similarity as the intra-sample feature matching degree of the training sample. For each intra-sample feature matching degree, the negative of the logarithm of the intra-sample feature matching degree with a preset exponent as the base is used as the cross entropy of the intra-sample feature matching degree. The cross entropies of the intra-sample feature matching degrees are fused to determine the positive example matching loss. For each training sample, the model provider calculates the similarity between the image feature of the training sample and the text feature of other training samples except the training sample to obtain the inter-sample feature matching degree of the training sample. For each feature matching degree between samples, the difference between the unit value and the feature matching degree between the samples is calculated, and the model provider uses the negative of the logarithm of the difference with a preset exponent as the cross entropy of the feature matching degree between the samples. The cross entropies of the feature matching degrees between the samples are fused to obtain the negative example matching loss. The model provider fuses the sum of the cross entropies of the feature matching degrees within each sample in the positive example matching loss and the sum of the cross entropies of the feature matching degrees between each sample in the negative example matching loss to determine the matching loss function. The model provider updates the network parameters of the image feature extraction model and the text feature extraction model with the goal of minimizing the matching loss function, and then returns to the step of obtaining multiple training samples to continue training until the training stop condition is met, and a general image feature extraction model is obtained.

[0214] Step 2: The computer device in the universal image feature system obtains the universal image feature extraction model from the model provider. When the computer device receives the feature query request with the target image identifier sent by the business party for the first time, the computer device returns information to the business party that there is no universal image feature corresponding to the target image identifier to remind the business party. The computer device receives the feature extraction request with the target image identifier sent by the business party, and the computer device queries the target image from the image library and obtains the target image corresponding to the target image identifier according to the target image identifier in the feature extraction request. The computer device calls the universal image feature extraction model to extract features from the target image and obtains the universal image features corresponding to the target image identifier. The computer device stores the target image and the universal image features corresponding to the target image identifier in the universal image feature library, and returns the universal image features corresponding to the target image identifier to the business party. Based on the target image identifier and the universal image features corresponding thereto, the computer device updates the mapping relationship between the image identifier and the universal image features to obtain an updated mapping relationship.

[0215] Step 3: When the computer device receives a feature query request carrying the target image identifier sent by any business party again, the computer device uses the updated mapping relationship as the mapping relationship corresponding to the feature query request carrying the target image identifier received again. The computer device queries whether there is a universal image feature corresponding to the target image identifier in the universal image feature library according to the mapping relationship corresponding to the feature query request carrying the target image identifier received again. If it is found that there is a universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, an error code is returned to the business party to indicate that the business party initiates a feature acquisition request. The computer device receives the feature acquisition request carrying the target image identifier sent by the business party, and according to the feature acquisition request, directly returns the universal image feature corresponding to the target image identifier obtained from the universal image feature library. If it is found that there is no universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, an abnormal prompt message is returned.

[0216] Step 4: When the computer device receives the updated universal image feature extraction model sent by the model provider, the computer device clears the universal image features cached in the universal image feature library. After clearing the universal image features stored in the universal image feature library, the computer device receives a feature query request carrying the target image identifier sent by the business party. If the computer device recognizes the target image identifier for the first time after the model is updated, the computer device determines that the feature query request carrying the target image identifier is received for the first time after the model is updated, and returns the updated universal image features of the target image obtained by calling the updated universal image feature extraction model to extract features from the target image corresponding to the target image identifier. The computer device stores the target image identifier and the updated universal image features of the target image, and based on the target image identifier and the corresponding updated universal image features, updates the mapping relationship between the image identifier and the updated universal image features. When the computer device receives a feature query request carrying the target image identifier sent by any business party again, the updated mapping relationship corresponding to the feature query request received for the first time is used as the mapping relationship corresponding to the feature query request carrying the target image identifier received again. The computer device determines whether there is an updated universal image feature corresponding to the target image identifier based on the mapping relationship corresponding to the feature query request carrying the target image identifier received again. If so, that is, the universal image identifier corresponding to the target image identifier is stored, the computer device directly returns the updated universal image feature corresponding to the target image identifier.

[0217] In this embodiment, since the general image feature extraction model is obtained by jointly training the image feature extraction model and the text feature extraction model, during the training process, the text information of the sample image is used as the supervision information, and the supervision information of the image in the specific business scenario is not used, so that the trained image feature extraction model has the ability to mine the common information of the target image applicable to different business parties. Therefore, when receiving the feature query request carrying the target image identifier sent by the business party, the general image feature of the target image obtained by calling the general image feature extraction model to extract the target image corresponding to the target image identifier is returned, and the target image identifier and the general image feature of the target image are stored in the general image feature library. In this way, when receiving the feature query request carrying the target image identifier sent by any business party again, the general image feature corresponding to the target image identifier is directly returned, avoiding repeated calculations of the same image by different business parties, and reducing the calculation consumption caused by repeated calculations. In addition, in the process of obtaining the general image feature, since the business party will not contact the original content medium (i.e., image or video), the original content medium is protected to a great extent, avoiding the risk of data leakage.

[0218] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0219] Based on the same inventive concept, the embodiment of the present application also provides an image feature acquisition device for implementing the above-mentioned image feature acquisition method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image feature acquisition device embodiments provided below can refer to the limitations of the image feature acquisition method above, and will not be repeated here.

[0220] In one embodiment, Fig.11 As shown, an image feature acquisition device 1100 is provided, including: a receiving module 1102 and a storage module 1104, wherein:

[0221] The receiving module 1102 is used to return the general image features of the target image obtained by calling the general image feature extraction model to extract features from the target image corresponding to the target image identifier when receiving a feature query request carrying the target image identifier sent by the business party; wherein the general image feature extraction model is obtained by jointly training the image feature extraction model and the text feature extraction model, each training sample includes a sample image and text information of the sample image, and the image feature extraction model is updated according to a matching loss function determined based on a plurality of the training samples, and the matching loss function is determined according to the matching degree between the image features obtained by extracting features from the sample image using the general image feature extraction model and the text features obtained by extracting features from the text information using the text feature extraction model;

[0222] The storage module 1104 is used to store the target image identifier and the universal image features of the target image in the universal image feature library so that when a feature query request carrying the target image identifier is received again from any business party, the universal image features corresponding to the target image identifier are directly returned.

[0223] In one embodiment, the receiving module 1102 is used to obtain a target image corresponding to the target image identifier based on the target image identifier when receiving a feature query request carrying a target image identifier sent by a business party, and call a universal image feature extraction model to extract features from the target image to obtain universal image features corresponding to the target image identifier, and return the universal image features corresponding to the target image identifier in response to the feature query request.

[0224] In one embodiment, the receiving module 1102 is used to query whether there is a universal image feature corresponding to the target image identifier in the universal image feature library based on the mapping relationship between the image identifier and the universal image feature when a feature query request carrying the target image identifier is received again from any business party; if it is found that there is a universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, the universal image feature corresponding to the target image identifier queried from the universal image feature library is returned; if it is found that there is no universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, an abnormal prompt message is returned.

[0225] In one embodiment, the image feature acquisition device 1100 further includes a clearing module 1106, and the clearing module 1106 is used to clear the general image features stored in the general image feature library when an updated general image feature extraction model is received.

[0226] In one embodiment, the receiving module 1102 is further used to return the updated universal image features of the target image obtained by calling the updated universal image feature extraction model to extract features from the target image corresponding to the target image identifier if a feature query request carrying the target image identifier is received from a business party; the storage module 1104 is further used to store the target image identifier and the updated universal image features of the target image in a universal image feature library in correspondence with each other, so as to directly return the updated universal image features corresponding to the target image identifier when a feature query request carrying the target image identifier is received from any business party again.

[0227] In one embodiment, the image feature acquisition device 1100 also includes a training module 1108, which is used to acquire multiple training samples, each training sample including a sample image and text information corresponding to the sample image; through an image feature extraction model, feature extraction is performed on the sample image of each training sample to obtain the image feature of each training sample; through a text feature extraction model, feature extraction is performed on the text information of each training sample to obtain the text feature of each training sample; a positive example matching loss is constructed based on the image features and text features of the same training sample; a negative example matching loss is constructed based on the image features and text features of each training sample; a matching loss function is determined based on the positive example matching loss and the negative example matching loss; based on the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained, and the trained image feature extraction model is used as a general image feature extraction model.

[0228] In one embodiment, the training module 1108 is used to calculate the similarity between the image feature and the text feature of each training sample, obtain the intra-sample feature matching degree of each training sample, and determine the positive example matching loss according to the intra-sample feature matching degree of each training sample; the training module 1108 is used to calculate the similarity between the image feature of the training sample and the text feature of each training sample other than the training sample, obtain the inter-sample feature matching degree of each training sample, and determine the negative example matching loss according to the inter-sample feature matching degree of each training sample; the training module 1108 is used to determine the matching loss function according to the intra-sample feature matching degree of each training sample and the inter-sample feature matching degree, and the matching loss function is inversely correlated with the intra-sample feature matching degree and positively correlated with the inter-sample feature matching degree.

[0229] In one embodiment, the training module 1108 is used to calculate the sum of the cross entropies of the feature matching degrees within each sample and the sum of the cross entropies of the feature matching degrees between each sample; and determine the matching loss function based on the sum of the cross entropies of the feature matching degrees within each sample and the sum of the cross entropies of the feature matching degrees between each sample.

[0230] In one embodiment, the training module 1108 is used to update the network parameters of the image feature extraction model and the text feature extraction model with the goal of minimizing the matching loss function, and then return to the step of obtaining multiple training samples to continue training until the training stop condition is met, thereby obtaining a general image feature extraction model.

[0231] Based on the same inventive concept, the embodiment of the present application also provides a training device for a general image feature extraction model for implementing the training method for the general image feature extraction model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more general image feature extraction models provided below can refer to the limitations of the training method for the general image feature extraction model above, and will not be repeated here.

[0232] In one embodiment, Fig.12 As shown, a training device 1200 for a general image feature extraction model is provided, comprising an acquisition module 1202, an extraction module 1204, a construction module 1206, a determination module 1208 and an update module 1210, wherein:

[0233] The acquisition module 1202 is used to acquire multiple training samples, each of which includes a sample image and text information corresponding to the sample image.

[0234] The extraction module 1204 is used to extract features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample, and to extract features from the text information of each training sample using a text feature extraction model to obtain text features of each training sample.

[0235] The construction module 1206 is used to construct a positive example matching loss based on the image features and text features of the same training sample, and to construct a negative example matching loss based on the image features and text features of each pair of training samples.

[0236] The determination module 1208 is used to determine the matching loss function according to the positive example matching loss and the negative example matching loss.

[0237] The updating module 1210 is used to jointly train the image feature extraction model and the text feature extraction model according to the matching loss function, and use the trained image feature extraction model as the general image feature extraction model.

[0238] Each module in the above-mentioned image feature acquisition device and the training device of the general image feature extraction model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0239] In one embodiment, a computer device is provided, which may be Figure 3 The terminal or server shown in FIG. 1 may have an internal structure as shown in FIG. Fig.13As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image feature acquisition method or a training method for a general image feature extraction model is implemented.

[0240] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0241] In one embodiment, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the image feature acquisition method or the training method of the general image feature extraction model described in the above-mentioned method embodiments are implemented.

[0242] In one embodiment, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the image feature acquisition method or the training method of the general image feature extraction model described in the above-mentioned method embodiments are implemented.

[0243] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image feature acquisition method or the training method of the general image feature extraction model described in the above method embodiments are implemented.

[0244] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the image feature acquisition method or the training method of the general image feature extraction model described in the above-mentioned method embodiments.

[0245] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0246] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0247] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0248] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for acquiring image features, characterized in that: Applied to a computer device for providing universal image features, the method comprises: Upon receiving a feature query request carrying a target image identifier sent by a business party, according to the target image identifier, a target image corresponding to the target image identifier is obtained from a local image library, the business party is a party using universal image features, and a universal image feature extraction model is obtained from a model provider, and features are extracted from the target image based on the universal image feature extraction model to obtain universal image features corresponding to the target image identifier, and in response to the feature query request, the universal image features corresponding to the target image identifier are returned, so that the business party can perform reasoning on the corresponding business based on the universal image features; Among them, the universal image features of the target image represent the common information required by multiple businesses, and the universal image features of the target image are applicable to multiple businesses; the training step of the universal image feature extraction model is performed by the model provider, and the training step includes: obtaining multiple training samples, each of the training samples includes a sample image and text information corresponding to the sample image; through the image feature extraction model, feature extraction is performed on the sample image of each of the training samples to obtain the image features of each of the training samples; through the text feature extraction model, feature extraction is performed on the text information of each of the training samples to obtain the text features of each of the training samples; for each training sample, the similarity between the image features and the text features of the training sample is calculated to obtain the intra-sample feature matching degree of each of the training samples, and the positive example matching loss is determined based on the cross entropy of the intra-sample feature matching degree of each training sample, and the positive example matching loss reflects the matching The difference between the matched sample image and the text information; for each training sample, calculate the similarity between the image feature of the training sample and the text feature of each training sample other than the training sample, obtain the sample feature matching degree of each training sample, and determine the negative example matching loss based on the cross entropy of the feature matching degree between each sample, wherein the negative example matching loss reflects the difference between the mismatched sample image and the text information; according to the positive example matching loss and the negative example matching loss, determine the matching loss function, wherein the positive example matching loss is positively correlated with the matching loss function, and the negative example matching loss is positively correlated with the matching loss function; according to the matching loss function, jointly train the image feature extraction model and the text feature extraction model, use the trained image feature extraction model as the general image feature extraction model, and use the text feature extraction model to assist in training the image feature extraction model; The target image identifier and the universal image feature of the target image are stored in a universal image feature library in correspondence, so that when a feature query request carrying the target image identifier sent by other business parties is received again, the universal image feature corresponding to the target image identifier is directly returned; After the general image feature is cached in the general image feature library for a preset time period, the general image feature is cleared.

2. The method according to claim 1, characterized in that When receiving the feature query request carrying the target image identifier sent by other business parties again, directly returning the general image feature corresponding to the target image identifier includes: When receiving a feature query request carrying the target image identifier from another business party again, querying whether there is a universal image feature corresponding to the target image identifier in the universal image feature library according to the mapping relationship between the image identifier and the universal image feature; If it is found that the universal image feature library has a universal image feature corresponding to the target image identifier, in response to the feature query request, the universal image feature corresponding to the target image identifier that is found from the universal image feature library is returned; If it is found that the universal image feature library does not contain the universal image feature corresponding to the target image identifier, an abnormal prompt information is returned in response to the feature query request.

3. The method according to claim 1, characterized in that The method further comprises: When an updated universal image feature extraction model is received, the universal image features stored in the universal image feature library are cleared.

4. The method according to claim 3, characterized in that The method further comprises: If a feature query request carrying a target image identifier is received from a business party, an updated universal image feature of the target image obtained by extracting features from the target image corresponding to the target image identifier by calling an updated universal image feature extraction model is returned; The target image identifier and the updated universal image features of the target image are stored in a universal image feature library in correspondence, so that when a feature query request carrying the target image identifier is received again from other business parties, the updated universal image features corresponding to the target image identifier are directly returned.

5. The method according to claim 1, characterized in that The specific steps of determining the matching loss function include: The sum of the cross entropies of the feature matching degrees within each sample is calculated, and the sum of the cross entropies of the feature matching degrees between each sample is calculated; and the matching loss function is determined according to the sum of the cross entropies of the feature matching degrees within each sample and the sum of the cross entropies of the feature matching degrees between each sample.

6. The method according to claim 1, characterized in that The method of jointly training the image feature extraction model and the text feature extraction model according to the matching loss function and using the trained image feature extraction model as the general image feature extraction model includes: With the goal of minimizing the matching loss function, after updating the network parameters of the image feature extraction model and the text feature extraction model, return to the step of obtaining multiple training samples to continue training until the training stop condition is met, thereby obtaining the universal image feature extraction model.

7. A training method for a general image feature extraction model, characterized in that: Applied to the model provider, the method includes: Acquire a plurality of training samples, each of the training samples comprising a sample image and text information corresponding to the sample image; Extracting features from the sample images of each training sample using an image feature extraction model to obtain image features of each training sample; By using a text feature extraction model, feature extraction is performed on the text information of each training sample to obtain text features of each training sample; For each training sample, the similarity between the image feature and the text feature of the training sample is calculated to obtain the in-sample feature matching degree of each training sample, and the positive example matching loss is determined based on the cross entropy of the in-sample feature matching degree of each training sample, wherein the positive example matching loss reflects the degree of difference between the matched sample image and text information; For each training sample, the similarity between the image feature of the training sample and the text features of each training sample other than the training sample is calculated to obtain the inter-sample feature matching degree of each training sample, and the negative example matching loss is determined based on the cross entropy of the inter-sample feature matching degree, wherein the negative example matching loss reflects the difference between the mismatched sample image and text information; Determine a matching loss function according to the positive example matching loss and the negative example matching loss, wherein the positive example matching loss is positively correlated with the matching loss function, and the negative example matching loss is positively correlated with the matching loss function; According to the matching loss function, the image feature extraction model and the text feature extraction model are jointly trained, and the trained image feature extraction model is used as a general image feature extraction model, and the text feature extraction model is used to assist in training the image feature extraction model.

8. An image feature acquisition device, characterized in that: The device comprises: A receiving module is used to, when receiving a feature query request carrying a target image identifier sent by a business party, obtain a target image corresponding to the target image identifier from a local image library according to the target image identifier, wherein the business party is a party using universal image features, and obtains a universal image feature extraction model from a model provider, and performs feature extraction on the target image based on the universal image feature extraction model to obtain universal image features corresponding to the target image identifier, and returns the universal image features corresponding to the target image identifier in response to the feature query request, so that the business party can perform reasoning on the corresponding business based on the universal image features; wherein the universal image features of the target image represent common information required by multiple businesses, and the universal image features of the target image are applicable to multiple businesses; the device also includes a training module, which is used to obtain multiple training samples, each of which includes a sample image and text information corresponding to the sample image; through the image feature extraction model, feature extraction is performed on the sample image of each of the training samples to obtain the image features of each of the training samples; through the text feature extraction model, feature extraction is performed on the text information of each of the training samples to obtain the text features of each of the training samples The invention relates to a text feature of a sample; for each training sample, calculating the similarity between the image feature and the text feature of the training sample, obtaining the intra-sample feature matching degree of each training sample, and determining the positive example matching loss based on the cross entropy of the intra-sample feature matching degree of each training sample, wherein the positive example matching loss reflects the degree of difference between the matched sample image and the text information; for each training sample, calculating the similarity between the image feature of the training sample and the text feature of each training sample other than the training sample, obtaining the inter-sample feature matching degree of each training sample, and determining the negative example matching loss based on the cross entropy of the inter-sample feature matching degree, wherein the negative example matching loss reflects the degree of difference between the unmatched sample image and the text information; determining a matching loss function based on the positive example matching loss and the negative example matching loss, wherein the positive example matching loss is positively correlated with the matching loss function, and the negative example matching loss is positively correlated with the matching loss function; and jointly training the image feature extraction model and the text feature extraction model based on the matching loss function, and using the trained image feature extraction model as the general image feature extraction model, and the text feature extraction model is used to assist in training the image feature extraction model; A storage module, used to store the target image identifier and the universal image feature of the target image in a universal image feature library so as to directly return the universal image feature corresponding to the target image identifier when receiving a feature query request carrying the target image identifier from another business party again; The clearing module is used to clear the general image feature after the general image feature has been cached in the general image feature library for a preset time period.

9. The device according to claim 8, characterized in that The receiving module is used to query whether there is a universal image feature corresponding to the target image identifier in the universal image feature library according to the mapping relationship between the image identifier and the universal image feature when receiving the feature query request carrying the target image identifier sent by other business parties again; if it is found that there is a universal image feature corresponding to the target image identifier in the universal image feature library, then in response to the feature query request, return the universal image feature corresponding to the target image identifier found from the universal image feature library; If it is found that the universal image feature library does not contain the universal image feature corresponding to the target image identifier, an abnormal prompt information is returned in response to the feature query request.

10. The device according to claim 8, characterized in that The clearing module is further used to clear the general image features stored in the general image feature library when an updated general image feature extraction model is received.

11. The device according to claim 10, characterized in that The receiving module is also used to, if a feature query request carrying the target image identifier is received from a business party, return the updated general image features of the target image obtained by calling the updated general image feature extraction model to extract features from the target image corresponding to the target image identifier; the storage module is also used to store the target image identifier and the updated general image features of the target image in a general image feature library in correspondence with each other, so as to directly return the updated general image features corresponding to the target image identifier when a feature query request carrying the target image identifier is received again from other business parties.

12. The device according to claim 8, characterized in that The training module is used to calculate the sum of the cross entropies of the feature matching degrees within each sample and the sum of the cross entropies of the feature matching degrees between each sample; and determine the matching loss function based on the sum of the cross entropies of the feature matching degrees within each sample and the sum of the cross entropies of the feature matching degrees between each sample.

13. The device according to claim 8, characterized in that The training module is used to update the network parameters of the image feature extraction model and the text feature extraction model with the goal of minimizing the matching loss function, and then return to the step of obtaining multiple training samples to continue training until the training stop condition is met, thereby obtaining the general image feature extraction model.

14. A training device for a general image feature extraction model, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of training samples, each of which includes a sample image and text information corresponding to the sample image; An extraction module is used to extract features from the sample images of each training sample through an image feature extraction model to obtain image features of each training sample; and to extract features from the text information of each training sample through a text feature extraction model to obtain text features of each training sample; A construction module is used to calculate, for each training sample, the similarity between the image feature and the text feature of the training sample, obtain the intra-sample feature matching degree of each training sample, determine the positive example matching loss based on the cross entropy of the intra-sample feature matching degree of each training sample, and the positive example matching loss reflects the degree of difference between the matched sample image and the text information; for each training sample, calculate the similarity between the image feature of the training sample and the text features of each training sample other than the training sample, obtain the inter-sample feature matching degree of each training sample, determine the negative example matching loss based on the cross entropy of the inter-sample feature matching degree, and the negative example matching loss reflects the degree of difference between the mismatched sample image and the text information; A determination module, configured to determine a matching loss function according to the positive example matching loss and the negative example matching loss, wherein the positive example matching loss is positively correlated with the matching loss function, and the negative example matching loss is positively correlated with the matching loss function; An updating module is used to jointly train the image feature extraction model and the text feature extraction model according to the matching loss function, and use the trained image feature extraction model as a general image feature extraction model, and the text feature extraction model is used to assist in training the image feature extraction model.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images