Data processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a computer readable storage medium. The method comprises the steps of obtaining a first feature library and an image recognition model; obtaining a second image containing a second object, and performing feature extraction on the second image through the image recognition model to obtain a second feature vector; and associatively storing the second feature vector and a second tag used for indicating a second object in the first feature library to obtain a second feature library. By adopting the method and the device, the model training cost can be reduced, and the generalization of the image recognition model can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the popularization of multimedia technology and the development of computer Internet, image applications can be seen everywhere. Among them, image recognition is one of the hottest areas of image application. Image recognition refers to identifying the target object in the image.

[0003] In the prior art, if object 1 (e.g., a cat) is used as the target object, an initial model is trained with a large number of sample images containing cats to obtain model 1 that can accurately identify images containing cats; subsequently, if object 1 (e.g., a dog) is used as another target object, an initial model is trained with a large number of sample images containing dogs to obtain model 2 that can accurately identify images containing dogs. Obviously, when a new object to be identified is added, the prior art needs to retrain the model, thereby increasing the model training cost. In addition, model 1 in the above example cannot identify images containing dogs, and model 2 cannot identify images containing cats, so the model constructed by the prior art has low generalization. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device, and computer-readable storage medium, which can not only reduce the model training cost, but also improve the generalization of the image recognition model.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Acquire a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label stored in association; the first feature vector is used to characterize a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image using the image recognition model;

[0007] Acquire a second image containing a second object, and extract features from the second image using an image recognition model to obtain a second feature vector; the second object is different from the first object;

[0008] The second feature vector and the second label indicating the second object are associated and stored in the first feature library to obtain a second feature library; the second feature library is used to determine together with the image recognition model that the first image to be detected includes a target object; the target object includes at least one of the first object or the second object.

[0009] On the one hand, an embodiment of the present application provides a data processing device, the device comprising:

[0010] An acquisition module is used to acquire a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label stored in association; the first feature vector is used to characterize a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image through the image recognition model;

[0011] an extraction module, configured to obtain a second image containing a second object, and extract features from the second image using an image recognition model to obtain a second feature vector; the second object is different from the first object;

[0012] A storage module is used to associate and store a second feature vector and a second label indicating a second object in a first feature library to obtain a second feature library; the second feature library is used to jointly determine with an image recognition model that the first image to be detected includes a target object; the target object includes at least one of the first object or the second object.

[0013] In a possible implementation, the acquisition module is further used to perform the following operations:

[0014] Obtain a first sample image in a test set, perform feature extraction on the first sample image through an image recognition model, and obtain a sample feature vector corresponding to the first sample image;

[0015] According to the sample feature vector and the first feature vector, a target similarity threshold corresponding to the first object is determined from at least two candidate similarity thresholds corresponding to the first object; the target similarity threshold is used together with the first feature vector and the image recognition model to determine that the second image to be detected includes the first object.

[0016] In a possible implementation, the acquisition module determines a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds corresponding to the first object according to the sample feature vector and the first feature vector, and performs the following operations:

[0017] Performing similarity processing on the sample feature vector and the first feature vector to obtain a first similarity value;

[0018] Comparing the first similarity value with at least two candidate similarity thresholds corresponding to the first object, respectively, to obtain image prediction results corresponding to the at least two candidate similarity thresholds;

[0019] Verifying at least two image prediction results by using the label carried by the first sample image to obtain image prediction verification results corresponding to the at least two image prediction results;

[0020] According to the at least two image prediction verification results, a target similarity threshold corresponding to the first object is determined from at least two candidate similarity thresholds corresponding to the first object.

[0021] In a possible implementation, the at least two candidate similarity thresholds corresponding to the first object of the acquisition module include a candidate similarity threshold C b , at least two image prediction results include a candidate similarity threshold C b The corresponding image prediction result A b , b is a positive integer, and b is less than or equal to the total number of at least two candidate similarity thresholds;

[0022] Verify the at least two image prediction results using the label carried by the first sample image to obtain image prediction verification results corresponding to the at least two image prediction results, which are used to perform the following operations:

[0023] If the label carried by the first sample image is a first label used to indicate that the first sample image contains a first object, then in the image prediction result A b Indicates that the first similarity value is greater than or equal to the candidate similarity threshold C b When the image prediction result is A, the correct image prediction result is determined as image prediction result A b Image prediction verification results;

[0024] If the label carried by the first sample image is an invalid label for indicating that the first sample image does not contain the first object, then in the image prediction result A b Indicates that the first similarity value is less than the candidate similarity threshold C b When the image prediction result is A, the correct image prediction result is determined as image prediction result A b Image prediction verification results;

[0025] If the label carried by the first sample image is the first label, then in the image prediction result A b Indicates that the first similarity value is less than the candidate similarity threshold C b When the image prediction error result is determined as the image prediction result A b Image prediction verification results;

[0026] If the label carried by the first sample image is an invalid label, then in the image prediction result A b Indicates that the first similarity value is greater than or equal to the candidate similarity threshold C b When the image prediction error result is determined as the image prediction result A b The image prediction verification results.

[0027] In a possible implementation, the acquisition module compares the first similarity value with at least two candidate similarity thresholds corresponding to the first object, respectively, to obtain image prediction results corresponding to the at least two candidate similarity thresholds, respectively, for performing the following operations:

[0028] The first similarity value is compared with the candidate similarity threshold C b If the first similarity value is greater than or equal to the candidate similarity threshold C b , then the positive result of image prediction is determined as the candidate similarity threshold C b The corresponding image prediction result; the positive image prediction result is used to predict that the first sample image contains the first object;

[0029] If the first similarity value is less than the candidate similarity threshold C b , then the negative result of image prediction is determined as the candidate similarity threshold C b The corresponding image prediction result; the image prediction negative result is used to predict that the first sample image does not contain the first object.

[0030] In a possible implementation, the at least two candidate similarity thresholds corresponding to the first object of the acquisition module include a candidate similarity threshold C b , at least two image prediction verification results include candidate similarity threshold C b Corresponding image prediction verification result D b , b is a positive integer, and b is less than or equal to the total number of at least two candidate similarity thresholds;

[0031] Determining a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds corresponding to the first object according to at least two image prediction verification results includes:

[0032] Verify the result based on the image prediction D b , determine the candidate similarity threshold C b a first accurate value for a first sample image;

[0033] Get the candidate similarity threshold C b a second accurate value for a second sample image; the second sample image refers to the remaining sample images in the test set except the first sample image;

[0034] Determine the candidate similarity threshold C based on the first accurate value, the second accurate value, and the total number of sample images in the test set. b a corresponding first evaluation value;

[0035] According to the first evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds, a target similarity threshold corresponding to the first object is determined from the at least two candidate similarity thresholds.

[0036] In a possible implementation, the acquisition module determines the candidate similarity threshold C according to the first accurate value, the second accurate value, and the total number of all sample images in the test set. b The corresponding first evaluation value is used to perform the following operations:

[0037] If the label carried by the first sample image is the first label, then the image prediction verification result D b , determine the candidate similarity threshold C b a first recall value for a first sample image;

[0038] Get the candidate similarity threshold C b a second recall value for a third sample image; the third sample image refers to a sample image in the test set that carries the first label except the first sample image;

[0039] Determine the candidate similarity threshold C based on the first accurate value, the second accurate value, and the total number of sample images in the test set. b The corresponding total accurate value;

[0040] Determine the candidate similarity threshold C based on the first recall value, the second recall value, and the total number of sample images carrying the first label in the test set. b The corresponding total recall value;

[0041] Perform weighted summation on the total accuracy and total recall values ​​to obtain the candidate similarity threshold C b The corresponding first evaluation value.

[0042] In a possible implementation, the acquisition module determines a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds according to first evaluation values ​​respectively corresponding to at least two candidate similarity thresholds, and performs the following operations:

[0043] Obtaining a maximum first evaluation value from the first evaluation values ​​respectively corresponding to at least two candidate similarity thresholds;

[0044] The candidate similarity threshold corresponding to the maximum first evaluation value among the at least two candidate similarity thresholds is determined as the target similarity threshold corresponding to the first object.

[0045] In a possible implementation, the acquisition module is further used to perform the following operations:

[0046] A sample video is obtained, and frame extraction processing is performed on the sample video to obtain E sample images belonging to the sample video; E is a positive integer; and the E sample images include the first sample image;

[0047] Add E sample images to the test set;

[0048] The acquisition module determines the target similarity threshold corresponding to the first object from at least two candidate similarity thresholds according to the first evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds, and performs the following operations:

[0049] Get E sample images for candidate similarity threshold C b The corresponding image prediction verification results;

[0050] The number of results corresponding to the correct image prediction results in the E image prediction verification results is compared with the candidate similarity threshold C b The corresponding F candidate frame thresholds are compared respectively to obtain the video prediction results corresponding to the F candidate frame thresholds respectively; F is a positive integer greater than 1;

[0051] By using the label carried by the sample video, the F video prediction results are respectively verified to obtain video prediction verification results corresponding to the F video prediction results; the label carried by the sample video is a first label used to indicate that the sample video contains the first object, or an invalid label used to indicate that the sample video does not contain the first object;

[0052] According to the F video prediction verification results, determine the second evaluation values ​​corresponding to the F candidate frame thresholds respectively; wherein one video prediction verification result is used to determine the second evaluation value corresponding to one candidate frame threshold;

[0053] According to the F second evaluation values ​​and the candidate similarity threshold C b Corresponding to the first evaluation value, a target similarity threshold corresponding to the first object is determined from at least two candidate similarity thresholds.

[0054] In a possible implementation, the module obtains F candidate frame thresholds including a candidate frame threshold G h , h is a positive integer, and h is less than or equal to F; the F second evaluation values ​​include the candidate frame threshold G h a corresponding second evaluation value;

[0055] According to the F second evaluation values ​​and the candidate similarity threshold C b The corresponding first evaluation value determines a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds, and is used to perform the following operations:

[0056] The threshold G for the candidate frame h The corresponding second evaluation value and candidate similarity threshold C b The corresponding first evaluation values ​​are weighted and summed to obtain the candidate similarity threshold C b And the candidate frame threshold G h The total evaluation value of the combined candidate parameter pairs;

[0057] will contain the candidate similarity threshold C b The total evaluation value corresponding to each of the F candidate parameter pairs is determined as the candidate similarity threshold C b The corresponding F total evaluation values;

[0058] Obtain F total evaluation values ​​corresponding to at least two candidate similarity thresholds respectively, and obtain a maximum total evaluation value among the F total evaluation values ​​corresponding to at least two candidate similarity thresholds respectively;

[0059] Determine the candidate similarity threshold corresponding to the maximum total evaluation value among at least two candidate similarity thresholds as the target similarity threshold corresponding to the first object;

[0060] The candidate frame threshold corresponding to the maximum evaluation value among at least two candidate frame thresholds is determined as the target frame threshold corresponding to the first object; the target parameter pair composed of the target similarity threshold and the target frame threshold is used together with the first feature vector and the image recognition model to jointly determine that the second image to be detected includes the first object.

[0061] In a possible implementation, the first total number of first feature vectors of the acquisition module is I; I is a positive integer greater than 1;

[0062] The sample feature vector and the first feature vector are processed for similarity to obtain a first similarity value, which is used to perform the following operations:

[0063] If I is greater than a first quantity threshold, the I first eigenvectors are divided to obtain a binary tree corresponding to the I first eigenvectors; the binary tree includes at least two leaf nodes; the at least two leaf nodes respectively correspond to eigenvector sets including the I first eigenvectors, and the eigenvectors respectively included in the at least two eigenvector sets are different from each other;

[0064] Among the at least two leaf nodes, determining a first leaf node that contains a sample feature vector;

[0065] According to the first leaf node, a candidate feature vector set is determined; the candidate feature vector set belongs to I first feature vectors; the candidate feature vector set includes j feature vectors; j is a positive integer greater than 1;

[0066] The sample feature vector and the j feature vectors are respectively processed for similarity to obtain j candidate similarity values, and the maximum candidate similarity value of the j candidate similarity values ​​is determined as the first similarity value.

[0067] In a possible implementation, the acquisition module determines a set of candidate feature vectors according to the first leaf node, and performs the following operations:

[0068] Among at least two leaf nodes, obtain a second leaf node adjacent to the first leaf node;

[0069] The feature vector set corresponding to the first leaf node and the feature vector set corresponding to the second leaf node are determined as candidate feature vector sets.

[0070] In a possible implementation, the acquisition module divides the I first eigenvectors to obtain a binary tree corresponding to the I first eigenvectors, which is used to perform the following operations:

[0071] Randomly select two eigenvectors from the I first eigenvectors as a first segmentation vector pair, use a hyperplane between the first segmentation vector pair as a segmentation plane, and perform segmentation processing on the I first eigenvectors to obtain two first intermediate eigenvector sets; the two first intermediate eigenvector sets include an intermediate eigenvector set k, and the intermediate eigenvector set k is any first intermediate eigenvector set in the two first intermediate eigenvector sets;

[0072] Determine a second total number of feature vectors in the intermediate feature vector set k, and if the second total number is greater than a second number threshold, randomly select two feature vectors in the intermediate feature vector set k as a second segmentation vector pair;

[0073] The hyperplane between the second segmentation vector pair is used as a segmentation plane, and the intermediate feature vector set k is divided to obtain two second intermediate feature vector sets; the two second intermediate feature vector sets include the intermediate feature vector set m, and the intermediate feature vector set m is any second intermediate feature vector set in the two second intermediate feature vector sets;

[0074] Determine a third total number of feature vectors in the intermediate feature vector set m, and if the mth total number is equal to or greater than the second number threshold, generate a leaf node for representing the intermediate feature vector set m;

[0075] According to the leaf nodes used to represent the intermediate feature vector set m, a binary tree corresponding to I first feature vectors is constructed.

[0076] In a possible implementation manner, the extraction module obtains a second image containing the second object, and is used to perform the following operations:

[0077] Acquire a target video containing a second object, and perform frame extraction processing on the target video to obtain at least two candidate images;

[0078] An image containing the second object is acquired from at least two candidate images, and the acquired image is determined as the second image.

[0079] In a possible implementation, the storage module is further configured to perform the following operations:

[0080] Determine the total number of first vectors of feature vectors in the second feature library; the feature vectors in the second feature library include the first feature vectors and the second feature vectors;

[0081] If the total number of the first vector is greater than the vector number threshold, then the feature vector N p Perform dimensionality reduction to obtain the feature vector N p The corresponding simplified eigenvector; eigenvector N p is any feature vector in the second feature library; feature vector N p The vector dimension of is greater than the vector dimension of the simplified eigenvector, and the simplified eigenvector contains the eigenvector N p key information.

[0082] In a possible implementation, the storage module is further configured to perform the following operations:

[0083] Determine the total number of first vectors of feature vectors in the second feature library; the feature vectors in the second feature library include first feature vectors and second feature vectors; the total number of second vectors of the first feature vectors is at least two;

[0084] If the total number of first vectors is greater than the vector number threshold, at least two first feature vectors are fused to obtain a synthetic feature vector for characterizing the first object.

[0085] In a possible implementation manner, the extraction module determines the acquired image as the second image, and performs the following operations:

[0086] The acquired image is determined as an initial image, and data enhancement processing is performed on the initial image to obtain an enhanced image different from the initial image;

[0087] The initial image and the enhanced image are determined as the second image.

[0088] In a possible implementation, the extraction module is further used to perform the following operations:

[0089] Acquire a first image to be detected, and extract features of the first image to be detected by using an image recognition model to obtain a third feature vector;

[0090] Performing similarity processing on the third eigenvector and the first eigenvector to obtain a second similarity value;

[0091] Performing similarity processing on the third eigenvector and the second eigenvector to obtain a third similarity value;

[0092] It is determined that the first image to be detected includes the target object according to the second similarity value and the third similarity value.

[0093] On one hand, the present application provides a computer device, including: a processor, a memory, and a network interface;

[0094] The above-mentioned processor is connected to the above-mentioned memory and the above-mentioned network interface, wherein the above-mentioned network interface is used to provide a data communication function, the above-mentioned memory is used to store a computer program, and the above-mentioned processor is used to call the above-mentioned computer program so that the computer device executes the method in the embodiment of the present application.

[0095] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the method in the embodiment of the present application.

[0096] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium; a processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method in the embodiment of the present application.

[0097] As can be seen from the above, when a second object different from the first object that has been added is added, the embodiment of the present application can extract features from the second image containing the second object through the image recognition model to obtain a second feature vector for characterizing the second image, add the second label for indicating the second object and the second feature vector to the first feature library to obtain the second feature library, and can determine that the image to be detected contains at least one of the first object or the second object through the second feature library and the image recognition model. Obviously, by constructing a feature library, the embodiment of the present application does not need to retrain the model when the number of objects to be recognized increases, and can use the existing model for reasoning, so the model training cost can be reduced, and since the feature vectors and labels in the feature library can be continuously enriched, the generalization of the image recognition model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0099] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0100] Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 1 ;

[0101] Figure 3 This is a data processing scenario provided by an embodiment of the present application. Figure 1 ;

[0102] Figure 4 This is a data processing scenario provided by an embodiment of the present application. Figure 2 ;

[0103] Figure 5 This is a data processing scenario provided by an embodiment of the present application. Figure 3 ;

[0104] Figure 6 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 2 ;

[0105] Figure 7 This is a data processing scenario provided by an embodiment of the present application. Figure 4 ;

[0106] Figure 8 This is a data processing scenario provided by an embodiment of the present application. Figure 5 ;

[0107] Fig. 9 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 3 ;

[0108] Fig.10 This is a data processing scenario provided by an embodiment of the present application. Figure 6 ;

[0109] Fig.11 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 4 ;

[0110] Fig.12 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0111] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0112] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0113] See also Figure 1 , Figure 1 Schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1 As shown, the system may include a business server 100 and a terminal device cluster. The terminal device cluster may include: terminal device 200a, terminal device 200b, terminal device 200c, ..., terminal device 200n. It can be understood that the above system may include one or more terminal devices, and this application does not limit the number of terminal devices.

[0114] There may be communication connections between the terminal device clusters, for example, there is a communication connection between the terminal device 200a and the terminal device 200b, and there is a communication connection between the terminal device 200a and the terminal device 200c. At the same time, any terminal device in the terminal device cluster may have a communication connection with the business server 100, for example, there is a communication connection between the terminal device 200a and the business server 100, wherein the above communication connection does not limit the connection method, and may be directly or indirectly connected by wired communication, directly or indirectly connected by wireless communication, or by other methods, and this application does not limit it here.

[0115] It should be understood that Figure 1 Each terminal device in the terminal device cluster shown in FIG. 1 may be installed with an application client. When the application client runs in each terminal device, it may be respectively connected to the above-mentioned Figure 1 The business server 100 shown in the figure performs data interaction, that is, the above-mentioned communication connection. Among them, the application client can be a video application, a convenient life application, a payment application, a financial application, a game application, a shopping application, a novel application, a camera application, a browser, etc., which has an object recognition function, and this application does not limit it here.

[0116] The application client may be an independent client or an embedded sub-client integrated in a certain client (e.g., a video client, an education client, a multimedia client, etc.), which is not limited here. Taking the video application as an example, the business server 100 may be a collection of multiple servers including a background server corresponding to the video application, a data processing server, etc. Therefore, each terminal device may transmit data with the business server 100 through the application client corresponding to the video application. For example, each terminal device may upload a first video containing a first object to the business server 100 through the application client of the video application, and then the business server 100 may recognize and process the first video to obtain a first image containing the first object.

[0117] It is understandable that in the specific implementation of the present application, related data such as user information (such as the first image and the second image) is involved. When the embodiments in the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant local laws, regulations and standards.

[0118] To facilitate subsequent understanding and description, the embodiments of the present application can be Figure 1 An example of a terminal device is selected from the terminal device cluster shown, for example, terminal device 200a is used for description. When a second image carrying a second tag is acquired in the application client, and a tag adding instruction for the second tag is received, the terminal device 200a can generate a tag adding request corresponding to the tag adding instruction in the application client. The second tag is used to indicate that the second image contains a second object, and the tag adding request is used to request that the second tag be added to an existing feature library (for example, the first feature library described below).

[0119] Through the application client, the terminal device 200a can send a tag adding request to the business server 100. The business server 100 obtains the second image carrying the second tag according to the tag adding request. The embodiment of the present application does not limit the way in which the business server 100 obtains the second image, which can be determined according to the actual application scenario. One feasible way to obtain is that the tag adding request includes the second image carrying the second tag, so the business server 100 can obtain the second image carrying the second tag from the tag adding request; another feasible way to obtain is that the tag adding request includes the storage path of the second image, so the business server 100 can obtain the storage path from the tag adding request, access the storage path, and obtain the second image carrying the second tag.

[0120] According to the tag adding request, the business server 100 obtains the first feature library and the image recognition model. The embodiment of the present application does not limit the model structure of the image recognition model, and it only needs to have the function of recognizing images. It can be composed of any one or more neural network models, such as Convolutional Neural Networks (CNN), Residual Network (ResNet), High-Resolution Net (HRNet), Standardized Convolutional Network Extension (EfficientNet), Contrastive Language-Image Pretraining (CLIP) model, etc.

[0121] The first feature library includes a first feature vector and a first label that are stored in association, wherein the first feature vector is used to characterize a first image containing a first object, the first label is used to indicate the first object, and the first feature vector is obtained by extracting features from the first image using an image recognition model. It is understandable that the process of adding the first label by the business server 100 is the same as the process of adding the second label. The embodiment of the present application does not limit the content of the object (including the first object and the second object mentioned above), which can be determined according to the actual application scenario, including but not limited to pigs, horses, cars, and film and television characters with protected usage rights.

[0122] Furthermore, through the image recognition model, the business server 100 can extract features from the second image to obtain a second feature vector for indicating the second image; the second feature vector and the second tag for indicating the second object are associated and stored in the first feature library to obtain a second feature library. It can be understood that the above processing completes the addition of the second tag. Subsequently, if a third tag different from the first tag and the second tag is added, the process of adding the third tag by the business server 100 is the same as the process of adding the second tag described above. Among them, the third tag is used to indicate a third object different from the first object and the second object.

[0123] Subsequently, if the first image to be detected is obtained, the service server can determine that the first image to be detected contains the target object through the second feature library and the image recognition model; the target object includes at least one of the first object or the second object. The present embodiment of the application does not describe the process of determining that the first image to be detected contains the target object. Please refer to the following figure. Figure 2 The description in the corresponding embodiment.

[0124] Optionally, if the terminal device 200a stores the first feature library and the above-mentioned image recognition model locally, and the terminal device 200a has offline computing capabilities, then when receiving the above-mentioned tag adding instruction, the image recognition model can be run locally, that is, the second image is input into the image recognition model, and the second image is subjected to feature extraction through the image recognition model to obtain the second feature vector output by the image recognition model. The terminal device 200a associates and stores the second feature vector and the second label indicating the second object in the first feature library to obtain the second feature library.

[0125] Since training the image recognition model involves a large amount of offline calculations, the local image recognition model of the terminal device 200a can be sent to the terminal device 200a after training or updating by the business server 100.

[0126] As can be seen from the above, when a second object different from the first object that has been added is added, the embodiment of the present application can extract features from the second image containing the second object through the image recognition model to obtain a second feature vector for characterizing the second image, add the second label for indicating the second object and the second feature vector to the first feature library to obtain the second feature library, and can determine that the image to be detected contains at least one of the first object or the second object through the second feature library and the image recognition model. Obviously, by constructing a feature library, the embodiment of the present application does not need to retrain the model when the number of objects to be recognized increases, and can use the existing model for reasoning, so the model training cost can be reduced, and since the feature vectors and labels in the feature library can be continuously enriched, the generalization of the image recognition model can be improved.

[0127] It is understandable that the method provided in the embodiments of the present application can be executed by a computer device, which includes but is not limited to a terminal device or a business server. Among them, the business server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. Among them, the terminal device and the business server can be directly or indirectly connected by wire or wireless means, and the embodiments of the present application are not limited here.

[0128] For further information, see Figure 2 , Figure 2 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc. The embodiments of the present application can be applied to business scenarios such as image recognition scenarios, image recommendation scenarios, image distribution scenarios, video recognition scenarios, video recommendation scenarios, and video distribution scenarios. The specific business scenarios will not be listed one by one here.

[0129] The implementation process of the data processing scenario can be carried out in the business server, in the terminal device, or interactively in the terminal device and the business server, which is not limited here. Figure 1 For any terminal device in the terminal device cluster of the corresponding embodiment, the service server may be the above-mentioned Figure 1 The service server 100 of the corresponding embodiment. For the convenience of description and understanding, the embodiment of the present application is described by taking the interaction between the terminal device and the service server as an example.

[0130] For the sake of ease of description and understanding, the present application embodiment takes the method executed by a business server as an example for explanation, that is, the business server executes the data processing method as a computer device. Figure 2 As shown, the data processing method may at least include the following steps S101 to S103.

[0131] Step S101, obtaining a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label that are stored in association; the first feature vector is used to characterize a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image through the image recognition model.

[0132] Specifically, the embodiment of the present application does not limit the number of first objects and the number of first images, which can be determined according to the actual application scenario. To facilitate subsequent understanding and explanation, the embodiment of the present application takes the number of first objects as 2, that is, 2 different first objects, such as object a and object b, so the first image includes two types of images, one is an image including object a, and the other is an image including object b.

[0133] Please also see Figure 3 , Figure 3 This is a data processing scenario provided by an embodiment of the present application. Figure 1 .like Figure 3 As shown, the number of the first objects is 2, namely a person and a cat. The first image includes an image containing a person and an image containing a cat, such as Figure 3 In the example, the first image includes image 31a containing a cat, image 32a containing a cat, image 33a containing a cat, image 31b containing a person, image 32b containing a person, and image 33b containing a person. It can be understood that the first tag is used to indicate the first object, corresponding to the first object, Figure 3 The example first tags include a tag 30a for indicating a cat (the tag can be understood as a text, the text is a cat), and a tag 30b for indicating a person (the tag can be understood as a text, the text is a person).

[0134] The business server obtains the image recognition model 30c. The embodiment of the present application does not limit the model structure of the image recognition model 30c, which can be composed of one or more neural network models and has the function of recognizing objects in the image.

[0135] The service server inputs the first image into the image generation model 30c. Figure 3 Taking image 31b as an example, it can be understood that the processing of the remaining images in the first image is similar to Figure 3The processing of image 31b in FIG. 3 is the same, so it will not be described here one by one. Figure 3 , the service server inputs the image 31b into the image recognition model 30c, and performs recognition processing on the image 31b through the image recognition model 30c, that is, performs feature extraction processing, and obtains the first feature vector corresponding to the image 31b, that is, Figure 3 The feature vector in 31d.

[0136] Further, the service server uses the first feature vector (i.e. Figure 3 The feature vector 31d in the image 31b and the label 30b used to indicate the object (i.e., person) contained in the image 31b are stored in association in the first feature library 30e. The associated storage between the feature vector 31d and the label 30b can be to construct an index relationship between the feature vector 31d and the label 30b, for example, to construct a pair of index key-value pairs by taking the feature vector 31d as the index key and the label 30b as the index value.

[0137] Step S102, obtaining a second image containing a second object, and extracting features from the second image using an image recognition model to obtain a second feature vector; the second object is different from the first object;

[0138] Specifically, a target video containing the second object is obtained, and frame extraction is performed on the target video to obtain at least two candidate images; an image containing the second object is obtained from the at least two candidate images, and the obtained image is determined as the second image.

[0139] Among them, the specific process of determining the acquired image as the second image may include: determining the acquired image as the initial image, performing data enhancement processing on the initial image to obtain an enhanced image different from the initial image; and determining the initial image and the enhanced image as the second image.

[0140] The embodiment of the present application does not limit the number of second objects and the number of second images, which can be determined according to actual application scenarios. For ease of subsequent understanding and explanation, the embodiment of the present application illustrates the number of second objects as 1, for example, object c is different from object a and object b.

[0141] The second image may come from a video, and the embodiment of the present application refers to the video containing the second image as a target video. Similarly, the embodiment of the present application does not limit the number of target videos, which may be one or more. It is understood that if the number of target videos is multiple, the processing processes corresponding to the multiple target videos are the same. The embodiment of the present application does not limit the source of the target video, which may come from a native video or from a recreated video.

[0142] Please also see Figure 4 , Figure 4 This is a data processing scenario provided by an embodiment of the present application. Figure 2 .like Figure 4 As shown, the service server obtains the target video 40a, Figure 4 The exemplary second object is a flower, that is, the second tag 40b carried by the target video 40a is a flower.

[0143] The service server extracts frames from the target video 40a to obtain at least two candidate images. The number of candidate images in the embodiment of the present application is 4. Figure 4 The examples are image 41c, image 42c, image 43c, and image 44c. Since the second object may appear in only individual frames or partial segments in the target video 40a, some of the acquired candidate images may contain the second object, while others may not. Figure 4 In the example shown, among the four candidate images, image 41c and image 42c are images containing flowers, and image 43c and image 44c are images not containing flowers.

[0144] The service server selects the image 41c containing flowers and the image 42c containing flowers from the four candidate images, and determines the image 41c containing flowers and the image 42c containing flowers as the second images.

[0145] It is understandable that in order to pursue diversity in form (including different styles, resolutions and deformations) and to improve the generalization of the image recognition model to the characteristics of the second object, the embodiment of the present application adopts data enhancement technology, including but not limited to rotation, scaling, color transformation, etc., to expand the diversity of data and capture the different characteristics of the second object.

[0146] Please see again Figure 4 , the business server uses image 41c and image 42c as initial images and processes the initial images using data enhancement technology. Figure 4 Taking image 41c as an example, it can be understood that the processing of image 42c in the initial image is similar to Figure 4 The processing process of image 41c in is the same, so it will not be repeated here.

[0147] Please see again Figure 4, the business server performs data enhancement processing on image 41c to obtain an enhanced image different from image 41c. The embodiment of the present application does not limit the specific method of data enhancement processing, which can be determined according to the actual application scenario. One feasible processing method is that the business server rotates the second object in image 41c, such as rotating the flower ninety degrees, to obtain enhanced image 41d; another feasible processing method is that the business server scales or expands the second object in image 41c, such as reducing the flower to one-fourth of its original size, to obtain enhanced image 42d; the third feasible processing method is that the business server changes the color of the second object in image 41c, such as changing the color of the flower to gray, to obtain enhanced image 43d.

[0148] It is understandable that the service server may determine image 41c and image 42c as the second image, and may also determine image 41c, image 42c, enhanced image 41d, enhanced image 42d, enhanced image 43d and an enhanced image corresponding to image 42c as the second image.

[0149] The process of the business server generating the second feature vector through the image recognition model is the same as the process of generating the first feature vector through the image recognition model in step S101, so please refer to the description of step S101 above and it will not be repeated here.

[0150] Step S103, the second feature vector and the second label for indicating the second object are associated and stored in the first feature library to obtain a second feature library; the second feature library is used to determine together with the image recognition model that the first image to be detected includes the target object; the target object includes at least one of the first object or the second object.

[0151] Specifically, a first image to be detected is obtained, and features are extracted from the first image to be detected through an image recognition model to obtain a third feature vector; similarity processing is performed on the third feature vector and the first feature vector to obtain a second similarity value; similarity processing is performed on the third feature vector and the second feature vector to obtain a third similarity value; and according to the second similarity value and the third similarity value, it is determined that the first image to be detected includes the target object.

[0152] The embodiment of the present application does not limit the number of first images to be detected, and can be determined according to the actual scenario. For the convenience of subsequent understanding and explanation, the embodiment of the present application illustrates the number of first images to be detected as 1. It is understandable that if the number of first images to be detected is multiple, the service server processes the multiple first images to be detected in the same manner.

[0153] Please also see Figure 3 , Figure 4 as well as Figure 5 , Figure 5This is a data processing scenario provided by an embodiment of the present application. Figure 3 . Figure 3 The number of the first object in the example is 2, namely a person and a cat. Figure 4 The number of the second object is 1, and the specific example is a flower. Figure 3 as well as Figure 4 , Figure 5 The exemplary second feature library 50d includes two first tags (eg Figure 5 The tag 30b for representing a person and the tag 30a for representing a cat) and a second tag (such as Figure 5 The tag 40b in the figure is used to characterize the flower.

[0154] As described in step S101 above, the second feature library 50d stores a first feature vector having an association relationship (specifically, an index relationship) with the tag 30b, such as Figure 5 The feature vector 31d (which can represent the feature vector 31d associated with the label 30b) shown in the example Figure 3 The image 31b of the example, the feature vector 32d associated with the label 30b (which can represent Figure 3 The image 32b of the example, the feature vector 33d associated with the label 30b (which can represent Figure 3 Similarly, the second feature library 50d stores a first feature vector having an association relationship (specifically, an index relationship) with the label 30a, such as Figure 5 The feature vector 31f (which can represent the feature vector 31f associated with the label 30a) shown in the example Figure 3 The image 31a of the example, the feature vector 32f associated with the label 30b (which can represent Figure 3 The image 32a of the example, the feature vector 33f associated with the label 30b (which can represent Figure 3 Example image 33a).

[0155] As described above, the second feature library 50d stores the second feature vectors that have an association relationship (specifically, an index relationship) with the tag 40b, such as Figure 5 The feature vector 41e associated with the label 40b shown in the example (for example, it can represent Figure 4 The image 41c of the example, the feature vector 42e associated with the label 40b (which can represent Figure 3 Example image 42c).

[0156] In addition, the second feature library 50d may also include a similarity threshold value corresponding to each tag, such as Figure 5In the example, the second feature library 50d includes a similarity threshold 51c with a value of 0.94 corresponding to the tag 30b (used to characterize a person), a similarity threshold 52c with a value of 0.9 corresponding to the tag 30a (used to characterize a cat), and a similarity threshold 53c with a value of 0.85 corresponding to the tag 40b (used to characterize a flower). The present embodiment does not describe the process of determining the similarity threshold corresponding to each tag, please refer to the following Figure 6 The description in the corresponding embodiment, and the following Fig. 9 The description in the corresponding embodiment.

[0157] The following describes the actual application scenario of the second feature library 50d. Figure 5 As shown, the service server obtains the first image to be detected, such as Figure 5 The service server inputs the image 51a into the image recognition model 30c, and performs recognition processing on the image 51a through the image recognition model 30c, that is, performs feature extraction processing, and obtains the third feature vector corresponding to the image 51a, that is, Figure 5 The feature vector 52a in .

[0158] Further, the service server sends the third feature vector (such as Figure 5 The example feature vector 52a) is processed for similarity with each feature vector in the second feature library 50d (including the first feature vector and the second feature vector mentioned above). The embodiment of the present application does not limit the method of similarity processing, and can be set according to the actual application scenario, including but not limited to cosine similarity, Euclidean distance and Manhattan distance.

[0159] It can be understood that the process of the business server performing similarity processing on the third feature vector and each feature vector in the second feature library is the same, so Figure 5 The following description will be given by taking the feature vector 33d associated with the label 30b, the feature vector 33f associated with the label 30a, and the feature vector 42e associated with the label 40b as examples.

[0160] The service server performs similarity processing on the feature vector 52a and the feature vector 33d used to represent the image 51a to obtain a second similarity value, such as Figure 5 The example takes a similarity value 51b of 0.3; the service server performs similarity processing on the feature vector 52a and the feature vector 33f to obtain a second similarity value, such as Figure 5The example similarity value 52b is 0.25; the business server performs similarity processing on the feature vector 52a and the feature vector 42e to obtain a third similarity value, such as Figure 5 The similarity value 53b shown in the example is 0.9.

[0161] Since the similarity value 51b is generated based on the feature vector 33d associated with the label 30b, the business server compares the similarity value 51b with the similarity threshold 51c corresponding to the label 30b, that is, compares 0.3 with 0.94. Since the similarity value 51b with a value of 0.3 is less than the similarity threshold 51c with a value of 0.94, the business server can determine that the image 51a does not contain a person.

[0162] Since the similarity value 52b is generated based on the feature vector 33f associated with the label 30a, the business server compares the similarity value 52b with the similarity threshold 52c corresponding to the label 30a, that is, compares 0.25 with 0.9. Since the similarity value 52b with a value of 0.25 is less than the similarity threshold 52c with a value of 0.9, the business server can determine that the image 51a does not contain a cat.

[0163] Since the similarity value 53b is generated based on the feature vector 42e associated with the label 40b, the business server compares the similarity value 53b with the similarity threshold 53c corresponding to the label 40b, that is, compares 0.9 with 0.85. Since the similarity value 53b with a value of 0.9 is greater than the similarity threshold 53c with a value of 0.85, the business server can determine that the image 51a contains flowers.

[0164] In actual application scenarios, the first image to be detected can come from the video to be detected. Therefore, the present embodiment can not only detect whether the image contains the object represented by the label (including the first label and the second label) in the second feature library, but also detect whether the video to be detected contains the target object. Specifically, the video frame in the video to be detected is detected. If the video frame in the video to be detected contains the target object, it can be determined that the video to be detected contains the target object. If the video frame in the video to be detected does not contain the target object, it can be determined that the video to be detected does not contain the target object.

[0165] In order to improve the detection accuracy of the video to be detected, in addition to setting the similarity threshold for the video frame (i.e., image), a frame threshold can also be set. For example, the frame threshold is 5, and the video frames in the video to be detected are detected. If the video frame in the video to be detected contains the target object, and there are at least 5 video frames containing the target object, it can be determined that the video to be detected contains the target object. If the number of video frames containing the target object in the video to be detected is less than 5, it can be determined that the video to be detected does not contain the target object.

[0166] Video content review is one of the main ecological review tasks of video platforms, and the review of objects with protected usage rights (such as film and television characters, copyright intellectual property (IP)) is an important direction of video content review. Objects with protected usage rights refer to certain characters or images, whose usage rights belong to specific companies or platforms (hereinafter collectively referred to as holders). Without the permission of the holders, no one can use these characters or images in commercial activities. Therefore, the business server needs to detect the acquired video content to determine whether objects with protected usage rights appear in the video, so as to control the infringing creative content and prevent the theft of usage rights.

[0167] For example, sender 1 sends a video to be detected to a business server. Using an embodiment of the present application, the business server performs object detection on the video to be detected and determines that the video to be detected contains an object with protected usage rights; if sender 1 is the holder of the object with protected usage rights, the business server can determine that sender 1 and the video to be detected do not infringe; if sender 1 is not the holder of the object with protected usage rights, the business server can determine that sender 1 and the video to be detected do infringe, and at this time, the video to be detected needs to be controlled.

[0168] As can be seen from the above, this application proposes an algorithm for detecting objects with protected usage rights. The key technology is to extract frames from a video containing an object with protected usage rights, and extract features from the video using an image recognition model to obtain a first feature vector, and associate the first feature vector with a first label indicating an object with protected usage rights and store it in a feature library to obtain a first feature library.

[0169] Assume that the first feature library contains features of n objects with protected usage rights. When the number of objects with protected usage rights increases from n to n+1, the business server obtains a video containing the newly added objects with protected usage rights, performs frame extraction and feature extraction on the video to obtain a second feature vector, and associates the second feature vector with a second label used to indicate the newly added objects with protected usage rights and stores them in the first feature library to obtain a second feature library.

[0170] The business server obtains the video to be detected, extracts frames from the video to be detected, and uses the image recognition model to extract features to obtain a third feature vector. The business server compares the third feature vector with the first feature vector for similarity to obtain a second similarity value corresponding to the first feature vector; the business server compares the third feature vector with the second feature vector for similarity to obtain a third similarity value corresponding to the second feature vector. Based on the second similarity value and the third similarity value, the business server determines whether the video to be detected contains an object with protected usage rights, thereby realizing protection of the object with protected usage rights.

[0171] The above description is only a preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

[0172] As can be seen from the above, when a second object different from the first object that has been added is added, the embodiment of the present application can extract features from the second image containing the second object through the image recognition model to obtain a second feature vector for characterizing the second image, add the second label for indicating the second object and the second feature vector to the first feature library to obtain the second feature library, and can determine that the image to be detected contains at least one of the first object or the second object through the second feature library and the image recognition model. Obviously, by constructing a feature library, the embodiment of the present application does not need to retrain the model when the number of objects to be recognized increases, and can use the existing model for reasoning, so the model training cost can be reduced, and since the feature vectors and labels in the feature library can be continuously enriched, the generalization of the image recognition model can be improved.

[0173] See also Figure 6 , Figure 6 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 2 The method may be performed by a business server (eg, Figure 1 The service server 100 shown in the figure may also be executed by a terminal device (for example, the above Figure 1 The terminal device 200a) shown in the figure can also be executed by the service server and the terminal device interactively. For ease of understanding, the embodiment of the present application takes the method executed by the service server as an example for explanation, that is, the service server executes the data processing method as a computer device. Figure 6 As shown, the process of the data processing method includes steps S201 to S205.

[0174] Step S201, obtaining a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label that are stored in association; the first feature vector is used to characterize a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image through the image recognition model.

[0175] For the specific implementation process of step S201, please refer to the above Figure 2 The corresponding step S101 in the embodiment will not be described in detail here.

[0176] Step S202: obtain a first sample image in a test set, perform feature extraction on the first sample image through an image recognition model, and obtain a sample feature vector corresponding to the first sample image.

[0177] Specifically, the test set includes positive sample images and negative sample images. The positive sample images are sample images containing the first object, and the negative sample images are sample images that do not contain the first object. The embodiment of the present application does not limit the number of types of the first object and can be determined based on the actual application scenario.

[0178] For ease of understanding and description, the embodiment of the present application sets the number of types of the first object to 1, specifically, a person. In this case, the first label example is the text "person". Figure 7 , Figure 7 This is a data processing scenario provided by an embodiment of the present application. Figure 4 .like Figure 7 In the example, the test set 70z includes image 71a, image 72a, image 73a, image 71b, image 72b, and image 73b, wherein image 71a, image 72a, and image 73a are all images containing the first object ( Figure 7 The example is a positive sample image of a person, that is, image 71a, image 72a and image 73a all carry the first label 70a; image 71b, image 72b and image 73b are all negative sample images that do not contain the first object. Figure 7 Example image 71b is a negative sample image containing a car, and image 71b carries a label that the example is a car, image 72b is a negative sample image containing a flower, and image 72b carries a label that the example is a flower, and image 73b is a negative sample image containing a radish, and image 73b carries a label that the example is a radish.

[0179] The service server obtains a first sample image from the test set 70z, wherein the first sample image may be any image in the test set 70z. Figure 7The first sample image 71a is used as an example for description. The service server inputs the image 71a into the image recognition model 30c, and the image recognition model 30c performs recognition processing on the image 71a, that is, performs feature extraction processing, and obtains the sample feature vector corresponding to the image 71a, that is, Figure 7 It can be understood that the processing of the remaining images in the test set is similar to Figure 7 The processing process of image 71a in is the same, so it will not be repeated here.

[0180] Step S203, determining a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds corresponding to the first object according to the sample feature vector and the first feature vector; the target similarity threshold is used together with the first feature vector and the image recognition model to determine that the second image to be detected includes the first object.

[0181] Specifically, similarity processing is performed on the sample feature vector and the first feature vector to obtain a first similarity value; the first similarity value is compared with at least two candidate similarity thresholds corresponding to the first object to obtain image prediction results corresponding to the at least two candidate similarity thresholds; verification processing is performed on the at least two image prediction results through the label carried by the first sample image to obtain image prediction verification results corresponding to the at least two image prediction results; based on the at least two image prediction verification results, a target similarity threshold corresponding to the first object is determined from the at least two candidate similarity thresholds corresponding to the first object.

[0182] The at least two candidate similarity thresholds corresponding to the first object include candidate similarity threshold C b , at least two image prediction results include a candidate similarity threshold C b The corresponding image prediction result A b , b is a positive integer, and b is less than or equal to the total number of at least two candidate similarity thresholds; the specific process of verifying at least two image prediction results by using the label carried by the first sample image to obtain the image prediction verification results corresponding to the at least two image prediction results may include: if the label carried by the first sample image is a first label used to indicate that the first sample image contains a first object, then when the first similarity value is greater than or equal to the candidate similarity threshold C b When the image prediction result is A, the correct image prediction result is determined as image prediction result A b If the label carried by the first sample image is an invalid label indicating that the first sample image does not contain the first object, then when the first similarity value is less than the candidate similarity threshold C b When the image prediction result is A, the correct image prediction result is determined as image prediction result A bIf the label carried by the first sample image is the first label, then when the first similarity value is less than the candidate similarity threshold C b When the image prediction error result is determined as the image prediction result A b If the label carried by the first sample image is an invalid label, then when the first similarity value is greater than or equal to the candidate similarity threshold C b When the image prediction error result is determined as the image prediction result A b The image prediction verification results.

[0183] The specific process of comparing the first similarity value with at least two candidate similarity thresholds corresponding to the first object to obtain image prediction results corresponding to at least two candidate similarity thresholds may include: comparing the first similarity value with the candidate similarity thresholds C b If the first similarity value is greater than or equal to the candidate similarity threshold C b , then the positive result of image prediction is determined as the candidate similarity threshold C b The corresponding image prediction result; the positive image prediction result is used to predict that the first sample image contains the first object; if the first similarity value is less than the candidate similarity threshold C b , then the negative result of image prediction is determined as the candidate similarity threshold C b The corresponding image prediction result; the image prediction negative result is used to predict that the first sample image does not contain the first object.

[0184] The at least two candidate similarity thresholds corresponding to the first object include candidate similarity threshold C b , at least two image prediction verification results include candidate similarity threshold C b Corresponding image prediction verification result D b , b is a positive integer, and b is less than or equal to the total number of at least two candidate similarity thresholds; according to the at least two image prediction verification results, the specific process of determining the target similarity threshold corresponding to the first object from the at least two candidate similarity thresholds corresponding to the first object may include: according to the image prediction verification result D b , determine the candidate similarity threshold C b For the first accurate value of the first sample image; obtain the candidate similarity threshold C b The second accurate value for the second sample image; the second sample image refers to the remaining sample images in the test set except the first sample image; according to the first accurate value, the second accurate value and the total number of all sample images in the test set, the candidate similarity threshold C is determined b The corresponding first evaluation value; and determining the target similarity threshold corresponding to the first object from the at least two candidate similarity thresholds according to the first evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds.

[0185] According to the first accurate value, the second accurate value and the total number of sample images in the test set, the candidate similarity threshold C is determined. b The specific process of the corresponding first evaluation value may include: if the label carried by the first sample image is the first label, then according to the image prediction verification result D b , determine the candidate similarity threshold C b The first recall value for the first sample image; obtaining the candidate similarity threshold C b The second recall value for the third sample image; the third sample image refers to the sample image in the test set that carries the first label except the first sample image; the candidate similarity threshold C is determined based on the first accuracy value, the second accuracy value and the total number of all sample images in the test set. b Corresponding total accuracy value; Determine the candidate similarity threshold C according to the first recall value, the second recall value and the total number of sample images carrying the first label in the test set b The corresponding total recall value; the total accuracy value and the total recall value are weighted and summed to obtain the candidate similarity threshold C b The corresponding first evaluation value.

[0186] Among them, the specific process of determining the target similarity threshold corresponding to the first object from at least two candidate similarity thresholds according to the first evaluation values ​​respectively corresponding to at least two candidate similarity thresholds may include: obtaining the maximum first evaluation value from the first evaluation values ​​respectively corresponding to at least two candidate similarity thresholds; determining the candidate similarity threshold corresponding to the maximum first evaluation value among the at least two candidate similarity thresholds as the target similarity threshold corresponding to the first object.

[0187] Please also see Figure 7 as well as Figure 8 , Figure 8 This is a data processing scenario provided by an embodiment of the present application. Figure 5 . Figure 8 Take a person as an example to describe the first object, such as Figure 8 As shown, the first feature library 80e includes a label 30b for characterizing a person. The first feature library 80e stores a first feature vector that has an association relationship (specifically, an index relationship) with the label 30b, such as Figure 8 In the example shown in FIG. 1 , there are a feature vector 31d associated with the label 30b, a feature vector 32d associated with the label 30b, and a feature vector 33d associated with the label 30b.

[0188] The business server obtains at least two candidate similarity thresholds 80a corresponding to the tag 30b. The embodiment of the present application does not limit the number of candidate similarity thresholds, which can be determined according to the actual scenario. For the convenience of subsequent understanding and explanation, the embodiment of the present application takes the number of candidate similarity thresholds as 3. Figure 8 In the example, at least two candidate similarity thresholds 80a include candidate similarity threshold 81a with a value of 0.75, candidate similarity threshold 82a with a value of 0.84, and candidate similarity threshold 83a with a value of 0.9. This embodiment of the application does not describe the determination process of candidate similarity thresholds, which can be set according to actual application scenarios.

[0189] If the number of types of the first object is multiple, such as the above Figure 3 In the example, the first object includes people and cats, then at least two candidate similarity thresholds corresponding to each object (ie, each label) may be different. For example, the candidate similarity thresholds corresponding to the label person include 0.75, 0.84 and 0.9, and the candidate similarity thresholds corresponding to the label cat include 0.76, 0.87 and 0.93.

[0190] Figure 8 Taking the feature vector 33d as the first feature vector for description, it can be understood that the processing method of the remaining feature vectors in the first feature library 80e is the same as Figure 8 The processing process of the feature vector 33d in is the same, so it will not be repeated here. Figure 7 The sample feature vector corresponding to image 71a in (such as Figure 8 The feature vector 71c of the example is processed with the feature vector 33d in the first feature library 80e to obtain a first similarity value, such as Figure 8 The similarity value 81b in the example is 0.85. The embodiment of the present application does not limit the method of similarity processing, and it can be set according to the actual application scenario.

[0191] Further, the business server compares the similarity value 81b with the candidate similarity threshold 81a, that is, compares 0.85 with 0.75. Since the similarity value 81b of 0.85 is greater than the candidate similarity threshold 81a of 0.75, the business server predicts that image 71a contains a person for the candidate similarity threshold 81a, and determines the positive image prediction result (used to indicate that image 71a contains a person) as the image prediction result 81c corresponding to the candidate similarity threshold 81a.

[0192] Similarly, the business server compares the similarity value 81b with the candidate similarity threshold 82a, that is, 0.85 with 0.84. Since the similarity value 81b of 0.85 is greater than the candidate similarity threshold 82a of 0.84, the business server predicts that image 71a contains a person for the candidate similarity threshold 82a, and determines the positive image prediction result (used to indicate that image 71a contains a person) as the image prediction result 82c corresponding to the candidate similarity threshold 82a.

[0193] Similarly, the business server compares the similarity value 81b with the candidate similarity threshold 83a, that is, compares 0.85 with 0.9. Since the similarity value 81b of 0.85 is smaller than the candidate similarity threshold 83a of 0.9, the business server predicts that image 71a does not contain a person for the candidate similarity threshold 83a, and determines the negative image prediction result (used to indicate that image 71a does not contain a person) as the image prediction result 83c corresponding to the candidate similarity threshold 83a.

[0194] Further, the business server obtains Figure 7 The label carried by the image 71a in the image 71a is the first label 70a used to represent a person. The business server verifies the image prediction result 81c according to the first label 70a. Since the image prediction result 81c is a positive image prediction result, that is, the image 71a is predicted to contain a person according to the candidate similarity threshold 81a, and the first label 70a is a person, the image prediction result is verified to be correct, and the correct image prediction result is determined as the image prediction verification result corresponding to the image prediction result 81c.

[0195] Similarly, the business server verifies the image prediction result 82c according to the first label 70a. Since the image prediction result 82c is a positive image prediction result, that is, the image 71a is predicted to contain a person according to the candidate similarity threshold 82a, and the first label 70a is a person, the image prediction result is verified to be correct, and the correct image prediction result is determined as the image prediction verification result corresponding to the image prediction result 82c. Similarly, the business server verifies the image prediction result 83c according to the first label 70a. Since the image prediction result 83c is a negative image prediction result, that is, the image 71a is predicted not to contain a person according to the candidate similarity threshold 83a, and the first label 70a is a person, the image prediction result is verified to be wrong, and the wrong image prediction result is determined as the image prediction verification result corresponding to the image prediction result 83c.

[0196] The following describes the evaluation process of the candidate similarity threshold, that is, the process of determining the first evaluation value of the candidate similarity threshold. It can be understood that the evaluation processes corresponding to at least two candidate similarity thresholds are the same, so the following is Figure 8The candidate similarity threshold 82a in the example is described, and the processing process of the remaining candidate similarity thresholds among the at least two candidate similarity thresholds can be found in the description below, which is not repeated here.

[0197] For the candidate similarity threshold 82a, if the image prediction verification result of the first sample image is a correct image prediction result, the service server determines the first value (eg, 1) as the first sample image (eg, Figure 7 if the image prediction verification result of the first sample image is an image prediction error result, the business server determines the second value (eg, 0) as the first accurate value corresponding to the first sample image.

[0198] The business server determines the remaining images in the test set except the first sample image as the second sample images. It can be understood that the process of determining the image prediction verification result of the second sample image is the same as the process of determining the image prediction verification result of the first sample image, so it is not repeated here.

[0199] For the candidate similarity threshold 82a, if the image prediction verification result of the second sample image is a correct image prediction result, the service server determines the first value (eg, 1) as the second sample image (eg, Figure 7 if the image prediction verification result of the second sample image is an image prediction error result, the business server determines the second numerical value (for example, 0) as the second accurate value corresponding to the second sample image.

[0200] For ease of understanding, the following terms are explained first.

[0201] A True Positive (TP) is an image in the test set that is actually a positive sample image and is correctly predicted as a positive sample image. For example, the first sample image is a positive sample image, and its image prediction result is a positive image prediction result.

[0202] A false positive (FP) is an image in the test set that is actually a negative sample image but is mistakenly predicted to be a positive sample image. For example, the first sample image is a negative sample image, but its image prediction result is a positive image prediction result.

[0203] A false negative (FN) is an image in the test set that is actually a positive sample image but is mistakenly predicted to be a negative sample image. For example, the first sample image is a positive sample image, but its image prediction result is a negative image prediction result.

[0204] A true negative (TN) is an image in the test set that is actually a negative sample image and is correctly predicted to be a negative sample image. For example, the first sample image is a negative sample image, and its image prediction result is an image prediction negative result.

[0205] The number of images S in the example test set of the embodiment of the present application is 100. Assuming that the test set includes 60 positive sample images and 40 negative sample images, through the method described above, for the candidate similarity threshold 82a, the business server determines that the number of true positive examples is 50, that is, the value of TP is 50; determines that the number of false positive examples is 10, that is, the value of FP is 10; determines that the number of false negative examples is 10, that is, the value of FN is 10; determines that the number of true negative examples is 30, that is, the value of TN is 30.

[0206] The service server can calculate the total accuracy value for the candidate similarity threshold 82a by the following formula (1):

[0207] Total accuracy = (TP + TN) / S (1)

[0208] Through formula (1), the service server determines that the total accuracy value for the candidate similarity threshold 82a is 0.8.

[0209] Further, the business server describes the first sample image as an example positive sample image. For the candidate similarity threshold 82a, if the image prediction verification result is a correct image prediction result, the business server determines the first value (for example, 1) as the first sample image (for example Figure 7 The first recall value corresponding to the image 71a in the image prediction verification result is an image prediction error result, the business server determines the second value (for example, 0) as the first recall value corresponding to the first sample image.

[0210] The business server determines the remaining images in the positive sample images in the test set except the first sample image as the third sample images. It can be understood that the process of determining the image prediction verification result of the third sample image is the same as the process of determining the image prediction verification result of the first sample image, so it is not repeated here.

[0211] For the candidate similarity threshold 82a, if the image prediction verification result is a correct image prediction result, the service server determines the first value (eg, 1) as the third sample image (eg, Figure 7 The second recall value corresponding to the image 72a in the image prediction verification result is an image prediction error result, the business server determines the second numerical value (for example, 0) as the second recall value corresponding to the third sample image.

[0212] Assuming that the test set includes 60 positive samples, through the method described above, for the candidate similarity threshold 82a, the business server determines that the number of true positive samples is 50, that is, the value of TP is 50; and determines that the number of false negative samples is 10, that is, the value of FN is 10.

[0213] The service server can calculate the total recall value for the candidate similarity threshold 82a by the following formula (2):

[0214] Total recall = TP / (TP + FN) (2)

[0215] Through formula (2), the business server determines that the total recall value for the candidate similarity threshold 82a is 5 / 6, which is approximately 0.83.

[0216] Further, for the candidate similarity threshold 82a, the business server performs weighted summation processing on the total accuracy value and the total recall value to obtain the first evaluation value corresponding to the candidate similarity threshold 82a. It can be understood that the embodiment of the present application does not limit the weighting method of the total accuracy value and the total recall value, which can be determined according to the actual application scenario.

[0217] The weighted value of the total accuracy value in the example of the embodiment of the present application is 0.7, and the weighted value of the total recall value is 0.3. For the candidate similarity threshold 82a, the service server performs weighted summation processing on the total accuracy value and the total recall value, and obtains the first evaluation value corresponding to the candidate similarity threshold 82a as 0.81.

[0218] According to the above processing, the business server can obtain the first evaluation value corresponding to each candidate similarity threshold. The business server obtains the maximum first evaluation value from the first evaluation values ​​corresponding to the above three candidate similarity thresholds, and determines the candidate similarity threshold corresponding to the maximum first evaluation value as the target similarity threshold corresponding to the first object.

[0219] Step S204, obtaining a second image containing a second object, and extracting features from the second image using an image recognition model to obtain a second feature vector; the second object is different from the first object;

[0220] For the specific implementation process of step S204, please refer to the above Figure 2 The corresponding step S102 in the embodiment will not be described in detail here.

[0221] Step S205, the second feature vector and the second label indicating the second object are associated and stored in the first feature library to obtain a second feature library; the second feature library is used to determine together with the image recognition model that the first image to be detected includes the target object; the target object includes at least one of the first object or the second object.

[0222] Specifically, the total number of first vectors of feature vectors in the second feature library is determined; the feature vectors in the second feature library include the first feature vectors and the second feature vectors; if the total number of first vectors is greater than the vector number threshold, the feature vector N p Perform dimensionality reduction to obtain the feature vector N p The corresponding simplified eigenvector; eigenvector N p is any feature vector in the second feature library; feature vector N p The vector dimension of is greater than the vector dimension of the simplified eigenvector, and the simplified eigenvector contains the eigenvector N p key information.

[0223] Specifically, the total number of first vectors of feature vectors in the second feature library is determined; the feature vectors in the second feature library include first feature vectors and second feature vectors; the total number of second vectors of first feature vectors is at least two; if the total number of first vectors is greater than a vector number threshold, at least two first feature vectors are fused to obtain a synthetic feature vector for characterizing the first object.

[0224] As the number of objects to be identified continues to increase, the number of feature vectors in the second feature library continues to increase. As the number of feature vectors in the second feature library continues to increase, the storage cost of the second feature library will also continue to increase. When performing object recognition on the first image to be detected, the retrieval cost of the feature vectors in the second feature library will also continue to increase. Therefore, it is necessary to reduce the retrieval cost at this time.

[0225] The business server determines the total number of first vectors of feature vectors in the second feature library, and compares the total number of first vectors with a vector number threshold. The vector number threshold is a parameter that can be adjusted according to the actual application scenario. For example, the total number of first vectors is 100,000. If the vector number threshold is greater than or equal to 100,000, the feature vectors in the second feature library are not processed. If the vector number threshold is less than 100,000, the business server further processes the feature vectors in the second feature library to reduce storage costs and retrieval costs. Several methods for reducing storage costs and retrieval costs are described below.

[0226] A specific implementation process may be as follows: the service server obtains a feature vector N from the second feature library p , for the eigenvector N pThe embodiment of the present application does not limit the method of feature dimensionality reduction, and can be set according to the actual application scenario, including but not limited to principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), uniform manifold approximation and projection (UMAP) and other dimensionality reduction technologies. The business server compresses high-dimensional features into low-dimensional space through feature dimensionality reduction technology, eliminates redundant features while retaining the main information of the features, and significantly reduces the space required for feature storage.

[0227] The business server obtains the feature vector N p In this embodiment of the present application, the feature vector N p Examples are Figure 5 The first feature vector 33f in the example is described, and the vector dimension of the first feature vector 33f is 512 dimensions. It can be understood that the processing method of the business server for the remaining feature vectors in the second feature library is the same as the processing method of the first feature vector 33f, so it is not repeated here. The business server performs dimensionality reduction processing on the first feature vector 33f to obtain a simplified feature vector corresponding to the first feature vector 33f. The dimension of the simplified feature vector in the example of the embodiment of the present application is 256 dimensions.

[0228] Subsequently, if an object different from the first object and the second object (hereinafter referred to as the newly added object) is added, for example, the newly added object is a dog, the image containing the dog (hereinafter referred to as the newly added image) is subjected to feature extraction through the image recognition model to obtain a newly added feature vector corresponding to the dog, and the vector dimension of the newly added feature vector is the same as the vector dimension of the first feature vector 33f, both of which are 512 dimensions. Therefore, the business server performs dimensionality reduction processing on the newly added feature vector to obtain a simplified feature vector corresponding to the newly added feature vector, that is, a simplified feature vector for characterizing the newly added object, and stores the simplified feature vector for characterizing the newly added object and the newly added label for characterizing the newly added object (such as the dog in the above example) in the second feature library.

[0229] The business server obtains the first image to be detected, and extracts features of the first image to be detected through the image recognition model to obtain a third feature vector. It can be understood that the vector dimension of the third feature vector is the same as the vector dimension of the first feature vector, so the third feature vector is subjected to dimensionality reduction processing to obtain a simplified feature vector corresponding to the third feature vector. The business server performs similarity processing on the simplified feature vector corresponding to the third feature vector and the simplified feature vector corresponding to the first feature vector to obtain a second similarity value; the business server performs similarity processing on the simplified feature vector corresponding to the third feature vector and the simplified feature vector corresponding to the second feature vector to obtain a third similarity value. Based on the second similarity value and the third similarity value, the business server determines that the first image to be detected includes the target object.

[0230] Another specific implementation process may be as follows: the business server performs a fusion process on at least two first feature vectors. The embodiment of the present application does not limit the feature fusion method, and may be set according to the actual application scenario, including but not limited to weighted averaging or maximum pooling of similar features. The business server integrates at least two similar feature vectors into a unified synthetic feature vector through feature fusion technology, thereby reducing redundant features and improving overall expression capabilities.

[0231] The service server obtains at least two first feature vectors. In the embodiment of the present application, the number of first feature vectors in the second feature library is 3. For example, the three first feature vectors are Figure 5 It can be understood that the processing method of the service server for the remaining feature vectors in the second feature library is the same as the processing method for the three first feature vectors, so they are not described one by one here.

[0232] The service server performs fusion processing on the three first feature vectors to obtain a synthetic feature vector for characterizing the first object.

[0233] Another specific implementation process may be as follows: the business server obtains the first feature vector associated with the first label, and clusters the highly similar feature vectors in the first feature vector according to similarity. The embodiment of the present application does not limit the method of feature clustering, and can be set according to the actual application scenario, including but not limited to K-means clustering (K-means Clustering, referred to as K-means), density-based spatial clustering applications (with noise) (Density-BasedSpatial Clustering of Applications with Noise, referred to as DBSCAN) or hierarchical clustering. The business server groups the feature vectors in the second feature library through feature clustering technology, and merges highly similar features into a representative feature vector according to similarity, thereby reducing redundant features while retaining the representativeness of the feature library.

[0234] Example of the present application Figure 3 If the first image 31a, the first image 32a and the first image 33a in the second feature library have highly similar features, the first feature vector 31f corresponding to the first image 31a, the first feature vector 32f corresponding to the first image 32a and the first feature vector 33f corresponding to the first image 33a are merged into a cluster feature vector. It can be understood that the processing method of the business server for the remaining feature vectors in the second feature library is the same as the processing method of the first feature vector 31f, the first feature vector 32f and the first feature vector 33f, so they are not described one by one here.

[0235] Another specific implementation process may be as follows: the business server obtains at least two first feature vectors, calculates the similarity of each two first feature vectors, and performs deduplication processing on at least two first feature vectors based on the vector similarity value between each two feature vectors. The embodiment of the present application does not limit the method of feature deduplication, and can be set according to the actual application scenario, including but not limited to calculating cosine similarity or Euclidean distance. The business server compares the vector similarity between each two first feature vectors with the vector similarity threshold. If there are two first feature vectors whose vector similarity is greater than or equal to the vector similarity threshold, one of the two first feature vectors is removed.

[0236] The first eigenvector of the embodiment of the present application is: Figure 5 It can be understood that the processing method of the service server for the remaining feature vectors in the second feature library is the same as that for the first feature vector 31f, the first feature vector 32f and the first feature vector 33f, so they are not described one by one here.

[0237] As can be seen from the above, by constructing a feature library, the embodiment of the present application does not need to retrain the model when the number of objects to be identified increases, and the existing model can be used for reasoning, so the model training cost can be reduced. Since the feature vectors and labels in the feature library can be continuously enriched, the generalization of the image recognition model can be improved. In addition, the embodiment of the present application can determine the target similarity threshold from at least two candidate similarity thresholds through the image prediction results of the sample images in the test set, so the accuracy of the target similarity threshold can be improved. Since image recognition is performed based on the target similarity threshold with high accuracy, the accuracy of image recognition can be improved.

[0238] See also Fig. 9 , Fig. 9 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 .like Fig. 9 As shown, the process of the data processing method includes steps S301 to S306, and steps S301 to S306 are Figure 6 The corresponding embodiment is a specific embodiment of determining a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds according to first evaluation values ​​respectively corresponding to at least two candidate similarity thresholds.

[0239] Step S301, obtain a sample video, perform frame extraction processing on the sample video, obtain E sample images belonging to the sample video, and add the E sample images to a test set; E is a positive integer; the E sample images include the first sample image.

[0240] Specifically, the business server obtains sample videos, which include positive sample videos and negative sample videos. The positive sample video is a video containing the first object, and the negative sample video is a video not containing the first object. The embodiment of the present application does not limit the number of sample videos, which can be determined according to the actual application scenario. The embodiment of the present application also does not limit the source of the sample video, which can come from native videos or from re-created videos. The ratio of positive sample videos to negative sample videos in the sample video conforms to the ratio of positive sample videos to negative sample videos that appear during the daily operation of the platform providing video services.

[0241] The embodiment of the present application does not limit the number of sample videos, which can be one or more. It is understandable that if the number of sample videos is multiple, the processing processes corresponding to the multiple sample videos are the same. Fig.10 , Fig.10 This is a data processing scenario provided by an embodiment of the present application. Figure 6 .like Fig.10 As shown, the business server uses sample video 10a as an example of a positive sample video for description. Fig.10The example first object is a person, that is, the label carried by the sample video 10a is a first label 10b used to characterize a person.

[0242] The service server extracts frames from the sample video 10a to obtain at least two sample images. Since people may appear in only individual frames or partial segments of the sample video 10a, some of the at least two sample images may contain people and some may not contain people. Fig.10 In the example, the at least two sample images include image 11c, image 12c, image 13c, image 14c, image 15c, and image 16c, wherein image 11c, image 12c, and image 13c are all images containing the first object ( Fig.10 The example is a positive sample image of a person, that is, image 11c, image 12c and image 13c all carry the first label 10b. Image 14c, image 15c and image 16c are all negative sample images that do not contain the first object. Fig.10 Example image 14c is an image containing a car, and image 14c carries a label indicating that the example is a car. Image 15c is an image containing clouds, and image 15c carries a label indicating that the example is a cloud. Image 16c is an image containing flowers, and image 16c carries a label indicating that the example is a flower.

[0243] Step S302: Obtain E sample images for candidate similarity threshold C b The corresponding image prediction verification results.

[0244] Specifically, the embodiments of the present application are based on Figure 8 It is described by taking the candidate similarity threshold 82a in the example, and it can be understood that the processing method of the remaining candidate similarity thresholds among the at least two candidate similarity thresholds is the same as the processing method of the candidate similarity threshold 82a, so it is not repeated here one by one.

[0245] It can be understood that the process in which the service server generates the image prediction verification results corresponding to the candidate similarity threshold 82a for the E sample images is similar to the above process. Figure 6 The process of generating the image prediction verification result corresponding to the candidate similarity threshold 82a for the image 71a in step S203 of the corresponding embodiment is the same, so please refer to the above Figure 6 The description of step S203 in the corresponding embodiment is not repeated here.

[0246] Please see again Fig.10 , the business server obtains the image prediction verification results corresponding to the candidate similarity threshold 82a of the 6 sample images, such as Fig.10The image prediction verification result 11d of the example image 11c for the candidate similarity threshold 82a is a correct image prediction result, the image prediction verification result 12d of image 12c for the candidate similarity threshold 82a is a correct image prediction result, the image prediction verification result 13d of image 13c for the candidate similarity threshold 82a is an incorrect image prediction result, the image prediction verification result 14d of image 14c for the candidate similarity threshold 82a is an incorrect image prediction result, the image prediction verification result 15d of image 15c for the candidate similarity threshold 82a is a correct image prediction result, and the image prediction verification result 16d of image 16c for the candidate similarity threshold 82a is an incorrect image prediction result.

[0247] Step S303: compare the number of results corresponding to the correct image prediction results in the E image prediction verification results with the candidate similarity threshold C b The corresponding F candidate frame thresholds are compared respectively to obtain the video prediction results corresponding to the F candidate frame thresholds respectively; F is a positive integer greater than 1.

[0248] Specifically, the service server obtains F candidate frame thresholds corresponding to the candidate similarity threshold 82a. The embodiment of the present application does not limit the number of candidate frame thresholds, which can be determined according to the actual scenario. For the convenience of subsequent understanding and explanation, the embodiment of the present application exemplifies that the number of candidate frame thresholds corresponding to the candidate similarity threshold 82a is 3, such as Fig.10 In the example, the three candidate frame thresholds include a candidate frame threshold 11e with a value of 2, a candidate frame threshold 12e with a value of 3, and a candidate frame threshold 13e with a value of 4. The present embodiment does not describe the determination process of the candidate frame thresholds, and can be set according to actual application scenarios.

[0249] like Fig.10 As shown, the business server obtains 3 results corresponding to the correct image prediction results from the 6 image prediction verification results, that is, the value of the result number 10d is 3. Further, the business server compares the result number 10d with the value of 3 with the candidate frame threshold 11e with the value of 2. Since the result number 10d with the value of 3 is greater than the candidate frame threshold 11e with the value of 2, it can be predicted that the sample video 10a contains a person, and the positive video prediction result is determined as the video prediction result 11f.

[0250] Similarly, the business server compares the result quantity 10d with a value of 3 with the candidate frame threshold 12e with a value of 3. Since the result quantity 10d with a value of 3 is equal to the candidate frame threshold 12e with a value of 3, it can be predicted that the sample video 10a contains a person, and the positive video prediction result is determined as the video prediction result 12f. Similarly, the business server compares the result quantity 10d with a value of 3 with the candidate frame threshold 13e with a value of 4. Since the result quantity 10d with a value of 3 is less than the candidate frame threshold 13e with a value of 4, it can be predicted that the sample video 10a does not contain a person, and the negative video prediction result is determined as the video prediction result 13f.

[0251] Step S304, verify the F video prediction results respectively through the labels carried by the sample video to obtain video prediction verification results corresponding to the F video prediction results respectively; the label carried by the sample video is a first label used to indicate that the sample video contains the first object, or an invalid label used to indicate that the sample video does not contain the first object.

[0252] For details, please refer to Fig.10 , the label carried by the sample video 10a is the first label 10b used to characterize a person. The business server verifies the video prediction result 11f through the first label 10b. Since the video prediction result 11f is a positive video prediction result, that is, the sample video 10a is predicted to contain a person according to the candidate similarity threshold 82a, and the first label 10b is a person, the video prediction result is verified to be correct, and the correct video prediction result is determined as the video prediction verification result corresponding to the video prediction result 11f.

[0253] Similarly, the business server verifies the video prediction result 12f through the first tag 10b. Since the video prediction result 12f is a positive video prediction result, that is, the sample video 10a is predicted to contain a person according to the candidate similarity threshold 82a, and the first tag 10b is a person, the video prediction result is verified to be correct, and the correct video prediction result is determined as the video prediction verification result corresponding to the video prediction result 12f. Similarly, the business server verifies the video prediction result 13f through the first tag 10b. Since the video prediction result 13f is a negative video prediction result, that is, the sample video 10a is predicted to contain a person according to the candidate similarity threshold 82a, and the first tag 10b is a person, the video prediction result is verified to be correct, and the wrong video prediction result is determined as the video prediction verification result corresponding to the video prediction result 13f.

[0254] In a feasible implementation manner, the business server obtains a sample image containing the first object (carrying the first label) from the E sample images and the image prediction verification result is a correct image prediction result. The obtained sample image can be understood as a true example, for example Fig.10Example images 11c and 12c, determine the number of true positives, Fig.10 Example is 2.

[0255] Further, the service server compares the number of true positive examples in the E sample images with at least two candidate frame thresholds. This process is the same as the above process. Fig.10 The three candidate frame thresholds of the example are described. Specifically, the business server compares the number of true positive examples 2 with the candidate frame threshold 11e with a value of 2. Since the number of true positive examples 2 is equal to the candidate frame threshold 11e with a value of 2, it can be predicted that the sample video 10a contains a person, and the positive video prediction result is determined as the video prediction result 11f. Similarly, the business server compares the number of true positive examples 2 with the candidate frame threshold 12e with a value of 3. Since the number of true positive examples 2 is less than the candidate frame threshold 12e with a value of 3, it can be predicted that the sample video 10a does not contain a person, and the negative video prediction result is determined as the video prediction result 12f. Similarly, the business server compares the number of true positive examples 2 with the candidate frame threshold 13e with a value of 4. Since the number of true positive examples 2 is less than the candidate frame threshold 13e with a value of 4, it can be predicted that the sample video 10a does not contain a person, and the negative video prediction result is determined as the video prediction result 13f.

[0256] It is understandable that the subsequent processing is the same as described above and will not be described in detail below.

[0257] Step S305, determining second evaluation values ​​corresponding to F candidate frame thresholds respectively according to the F video prediction verification results; wherein one video prediction verification result is used to determine the second evaluation value corresponding to one candidate frame threshold.

[0258] Specifically, the following describes the evaluation process of the candidate frame threshold, that is, the process of determining the second evaluation value of the candidate frame threshold. It can be understood that the evaluation processes corresponding to the three candidate frame thresholds are the same, so the following is Fig.10 The example of the candidate frame threshold 12e in FIG. 1 is described, and the process of determining the second evaluation value corresponding to the candidate frame threshold 12e is the same as that in the above Figure 6 The process of determining the first evaluation value corresponding to the candidate similarity threshold 82a in step S203 of the corresponding embodiment is the same, and the processing process of the remaining candidate frame thresholds among the three candidate frame thresholds can be found in the above description. Figure 6 The description of step S203 in the corresponding embodiment is not repeated here.

[0259] Step S306: based on the F second evaluation values ​​and the candidate similarity threshold C b Corresponding to the first evaluation value, a target similarity threshold corresponding to the first object is determined from at least two candidate similarity thresholds.

[0260] Specifically, for the candidate frame threshold G h The corresponding second evaluation value and candidate similarity threshold C b The corresponding first evaluation values ​​are weighted and summed to obtain the candidate similarity threshold C b And the candidate frame threshold G h The total evaluation value of the combined candidate parameter pair; it will include the candidate similarity threshold C b The total evaluation value corresponding to each of the F candidate parameter pairs is determined as the candidate similarity threshold C b F total evaluation values ​​corresponding to at least two candidate similarity thresholds; obtaining F total evaluation values ​​corresponding to at least two candidate similarity thresholds, and obtaining the maximum total evaluation value among the F total evaluation values ​​corresponding to at least two candidate similarity thresholds; determining the candidate similarity threshold corresponding to the maximum total evaluation value among the at least two candidate similarity thresholds as the target similarity threshold corresponding to the first object;

[0261] The candidate frame threshold corresponding to the maximum total evaluation value among at least two candidate frame thresholds is determined as the target frame threshold corresponding to the first object; the target parameter pair composed of the target similarity threshold and the target frame threshold is used together with the first feature vector and the image recognition model to jointly determine that the second image to be detected includes the first object.

[0262] The following describes the evaluation process of the candidate parameter pairs, that is, the process of determining the total evaluation value of the candidate parameter pairs of the candidate frame threshold and the candidate similarity threshold combination. It can be understood that the evaluation processes corresponding to the multiple candidate parameter pairs are the same, so the following is based on Fig.10 The candidate frame threshold 12e in Figure 8 The example of the candidate parameter pair combined with the candidate similarity threshold 82a in the above embodiment is used to describe the processing process of the remaining candidate parameters in the multiple candidate parameter pairs. Please refer to the description below and will not be described in detail.

[0263] The service server performs weighted sum processing on the second evaluation value corresponding to the candidate frame threshold 12e and the first evaluation value corresponding to the candidate similarity threshold 82a. The embodiment of the present application does not limit the weighting method of the first evaluation value and the second evaluation value, which can be determined according to the actual application scenario. For the convenience of subsequent understanding and explanation, the embodiment of the present application sets the weighted value of the first evaluation value to 0.6 and the weighted value of the second evaluation value to 0.4.

[0264] The second evaluation value corresponding to the candidate frame threshold 12e in the example embodiment of the present application is 0.9. For the candidate parameter pair composed of the candidate frame threshold 12e and the candidate similarity threshold 82a, the business server weights the second evaluation value of 0.9 and 0.4 to obtain a second weighted evaluation value of 0.36, and weights the first evaluation value of 0.81 and 0.6 to obtain a first weighted evaluation value of 0.486. The second weighted evaluation value of 0.36 and the first weighted evaluation value of 0.486 are summed to obtain a total evaluation value of 0.846 for the candidate parameter pair composed of the candidate similarity threshold 82a and the candidate frame threshold 12e.

[0265] According to the above processing, the service server can obtain the total evaluation value corresponding to each candidate parameter pair. Since each candidate similarity threshold is combined with three candidate frame thresholds to form a candidate parameter pair, the number of candidate parameter pairs is 9.

[0266] The business server determines the total evaluation values ​​corresponding to the three candidate parameter pairs including the candidate similarity threshold 81a as the three total evaluation values ​​corresponding to the candidate similarity threshold 81a; the business server determines the total evaluation values ​​corresponding to the three candidate parameter pairs including the candidate similarity threshold 82a as the three total evaluation values ​​corresponding to the candidate similarity threshold 82a; the business server determines the total evaluation values ​​corresponding to the three candidate parameter pairs including the candidate similarity threshold 83a as the three total evaluation values ​​corresponding to the candidate similarity threshold 83a.

[0267] The business server obtains 3 total evaluation values ​​corresponding to the candidate similarity threshold 81a, 3 total evaluation values ​​corresponding to the candidate similarity threshold 82a, and 3 total evaluation values ​​corresponding to the candidate similarity threshold 83a. The business server obtains the maximum total evaluation value from the above 9 total evaluation values, determines the candidate similarity threshold corresponding to the maximum total evaluation value as the target similarity threshold corresponding to the first object; and determines the candidate frame threshold corresponding to the maximum total evaluation value as the target frame threshold corresponding to the first object.

[0268] The service server uses a target parameter pair composed of a target similarity threshold and a target frame threshold, a first feature vector, and an image recognition model to jointly determine whether the second image to be detected includes the first object.

[0269] As can be seen from the above, by constructing a feature library, the embodiment of the present application does not need to retrain the model when the number of objects to be identified increases, and the existing model can be used for reasoning, so the model training cost can be reduced. Since the feature vectors and labels in the feature library can be continuously enriched, the generalization of the image recognition model can be improved. In addition, the embodiment of the present application extracts frames from the sample video to obtain E sample images. Through the image prediction results of the E sample images and the video prediction results of the sample video, the target similarity threshold can be determined from at least two candidate similarity thresholds, and the target frame threshold can be determined from F candidate frame thresholds. Therefore, the accuracy of the target parameter pair composed of the target similarity threshold and the target frame threshold can be improved. Since video recognition is performed based on the high-accuracy target parameter pair, the accuracy of video recognition can be improved.

[0270] This application proposes an object recognition method that has strong scalability and can flexibly adapt to the increase or decrease of objects. Specifically, this application can be applied in the following real-world scenarios:

[0271] 1. Video review scenario. This application can perform frame extraction on the video content, that is, convert a video into multiple pictures, and store the picture information in the "feature library". For the video to be reviewed, this application performs frame extraction, feature extraction and other steps to obtain the feature vector to be reviewed, and compares the similarity between the feature vector to be reviewed and the feature vector in the "feature library" to obtain the similarity value, and compares the similarity value with the target similarity threshold to determine whether the target object appears in the video. Therefore, this application is used in the video review scenario to reduce the cost of manual review.

[0272] 2. Scenarios where objects change frequently. In real life, objects will continue to change over time, for example, new objects will gradually emerge. When faced with scenarios where objects change frequently, previous technical solutions must perform steps such as data processing, model training, and model online. The present application can use one image recognition model to implement multiple object detections, which can replace the numerous object detection models previously designed for each object, and can be flexibly expanded to new objects without training. When acquiring an image containing a new object, the present application only needs to perform data processing and inject the information of the new object into the "feature library", reducing the cost of algorithm development. The present application can reduce the cost of algorithm development while ensuring good detection results, and has good scalability.

[0273] 3. Image review scenario. Although this application is a protection algorithm for video objects, it essentially processes the information of the image to obtain image information features, and stores the image information features in a "feature library". The similarity recall strategy is also designed for image information features. Therefore, this application can naturally be extended to image review scenarios, further expanding its scope of application.

[0274] See also Fig.11 , Fig.11 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 The method may be performed by a business server (eg, Figure 1 The service server 100 shown in the figure may also be executed by a terminal device (for example, the above Figure 1 The terminal device 200a) shown in the figure can also be executed by the service server and the terminal device interactively. For ease of understanding, the embodiment of the present application takes the method executed by the service server as an example for explanation, that is, the service server executes the data processing method as a computer device. Fig.11 As shown, the process of the data processing method includes steps S401 to S408.

[0275] Step S401, obtaining a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label that are stored in association; the first feature vector is used to characterize a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image through the image recognition model.

[0276] For the specific implementation process of step S401, please refer to the above Figure 2 The corresponding step S101 in the embodiment will not be described in detail here.

[0277] Step S402: Obtain a first sample image in a test set, perform feature extraction on the first sample image through an image recognition model, and obtain a sample feature vector corresponding to the first sample image.

[0278] For the specific implementation process of step S402, please refer to the above Figure 6 The corresponding step S202 in the embodiment is not described in detail here.

[0279] Step S403: perform similarity processing on the sample feature vector and the first feature vector to obtain a first similarity value.

[0280] Specifically, the first total number of first feature vectors is I; I is a positive integer greater than 1; if I is greater than a first quantity threshold, the I first feature vectors are divided to obtain a binary tree corresponding to the I first feature vectors; the binary tree includes at least two leaf nodes; the feature vector sets corresponding to at least two leaf nodes respectively include I first feature vectors, and the feature vectors respectively included in at least two feature vector sets are different from each other; among the at least two leaf nodes, determine the first leaf node containing the sample feature vector; according to the first leaf node, determine the candidate feature vector set; the candidate feature vector set belongs to the I first feature vectors; the candidate feature vector set includes j feature vectors; j is a positive integer greater than 1; perform similarity processing on the sample feature vector and the j feature vectors respectively to obtain j candidate similarity values, and determine the maximum candidate similarity value of the j candidate similarity values ​​as the first similarity value.

[0281] Among them, the specific process of determining the candidate feature vector set based on the first leaf node includes: obtaining the second leaf node adjacent to the first leaf node among at least two leaf nodes; and determining the feature vector set corresponding to the first leaf node and the feature vector set corresponding to the second leaf node as the candidate feature vector set.

[0282] The specific process of dividing I first eigenvectors to obtain a binary tree corresponding to the I first eigenvectors includes: randomly selecting two eigenvectors from the I first eigenvectors as a first segmentation vector pair, taking a hyperplane between the first segmentation vector pair as a segmentation plane, dividing the I first eigenvectors to obtain two first intermediate eigenvector sets; the two first intermediate eigenvector sets include an intermediate eigenvector set k, and the intermediate eigenvector set k is any one of the two first intermediate eigenvector sets; determining a second total number of eigenvectors in the intermediate eigenvector set k, and if the second total number is greater than a second number threshold, then in the intermediate eigenvector set k Randomly select two feature vectors as the second segmentation vector pair; use the hyperplane between the second segmentation vector pair as the segmentation plane, divide the intermediate feature vector set k, and obtain two second intermediate feature vector sets; the two second intermediate feature vector sets include the intermediate feature vector set m, and the intermediate feature vector set m is any second intermediate feature vector set in the two second intermediate feature vector sets; determine the third total number of feature vectors in the intermediate feature vector set m, and if the mth total number is equal to or less than the second number threshold, generate a leaf node for representing the intermediate feature vector set m; and construct a binary tree corresponding to I first feature vectors according to the leaf nodes for representing the intermediate feature vector set m.

[0283] The business server obtains the first feature library. In the example of the embodiment of the present application, the first total number of first feature vectors in the first feature library is 1000, for example, feature vector 1, feature vector 2, feature vector 3, ..., feature vector 999, feature vector 1000.

[0284] In the example of the embodiment of the present application, the value of the first quantity threshold is 500, and the value of the second quantity threshold is 50. Since the first total number of first feature vectors in the first feature library is 1000, and the first total number of 1000 is greater than the first quantity threshold of 500, the business server divides and processes the 1000 first feature vectors in the first feature library.

[0285] The business server randomly selects two first feature vectors in the first feature library as the first segmentation vector pair. In the embodiment of the present application, the first segmentation vector pair is exemplified as feature vector 1 and feature vector 500. The business server uses the hyperplane between feature vector 1 and feature vector 500 as the segmentation plane, and divides the 1000 first feature vectors through the segmentation plane to obtain two first intermediate feature vector sets, namely, intermediate feature vector set k1 and intermediate feature vector set k2. In the embodiment of the present application, the second total number of first feature vectors in the intermediate feature vector set k1 is exemplified as 400, such as feature vector 1, feature vector 2, ..., feature vector 400; the second total number of first feature vectors in the intermediate feature vector set k2 is exemplified as 600, such as feature vector 401, feature vector 402, ..., feature vector 1000.

[0286] The embodiment of the present application is described by taking the intermediate feature vector set k1 as an example. It can be understood that the processing process of the intermediate feature vector set k2 is the same as the processing process of the intermediate feature vector set k1. The business server compares the second total number of the first feature vectors in the intermediate feature vector set k1 with the second quantity threshold. Since the second total number with a value of 400 is greater than the second quantity threshold with a value of 50, the intermediate feature vector set k1 is divided. The business server randomly selects two feature vectors in the intermediate feature vector set k1 as the second segmentation vector pair. The second segmentation vector pair in the embodiment of the present application is feature vector 1 and feature vector 300. The business server uses the hyperplane between feature vector 1 and feature vector 300 as the segmentation plane, and divides the intermediate feature vector set k1 through the segmentation plane to obtain two second intermediate feature vector sets, namely, the intermediate feature vector set m1 and the intermediate feature vector set m2. In the embodiment of the present application, the third total number of first feature vectors in the intermediate feature vector set m1 is exemplified as 130, such as feature vector 1, feature vector 2, ..., feature vector 130; and the third total number of first feature vectors in the intermediate feature vector set m2 is exemplified as 270, such as feature vector 131, feature vector 132, ..., feature vector 400.

[0287] The embodiment of the present application is described by taking the intermediate feature vector set m1 as an example. It can be understood that the processing process of the intermediate feature vector set m2 is the same as the processing process of the intermediate feature vector set m1. The business server compares the third total number of the first feature vectors in the intermediate feature vector set m1 with the second quantity threshold. Since the third total number with a value of 130 is greater than the second quantity threshold with a value of 50, the intermediate feature vector set m1 is divided. The business server randomly selects two feature vectors in the intermediate feature vector set m1 as the third segmentation vector pair. The third segmentation vector pair in the embodiment of the present application is feature vector 1 and feature vector 100. The business server uses the hyperplane between feature vector 1 and feature vector 100 as the segmentation plane, and divides the intermediate feature vector set m1 through the segmentation plane to obtain two third intermediate feature vector sets, namely, intermediate feature vector set n1 and intermediate feature vector set n2. In an embodiment of the present application, the fourth total number of first feature vectors in the intermediate feature vector set n1 is exemplified as 70, such as feature vector 1, feature vector 2,…, feature vector 70; and the fourth total number of first feature vectors in the intermediate feature vector set n2 is exemplified as 60, such as feature vector 71, feature vector 72,…, feature vector 130.

[0288] The embodiment of the present application is described by taking the intermediate feature vector set n1 as an example. It can be understood that the processing process of the intermediate feature vector set n2 is the same as the processing process of the intermediate feature vector set n1. The business server compares the fourth total number of the first feature vectors in the intermediate feature vector set n1 with the second quantity threshold. Since the fourth total number with a value of 70 is greater than the second quantity threshold with a value of 50, the intermediate feature vector set n1 is divided. The business server randomly selects two feature vectors in the intermediate feature vector set n1 as the fourth segmentation vector pair. The fourth segmentation vector pair in the embodiment of the present application is feature vector 1 and feature vector 35. The business server uses the hyperplane between feature vector 1 and feature vector 35 as the segmentation plane, and divides the intermediate feature vector set n1 through the segmentation plane to obtain two fourth intermediate feature vector sets, namely, the intermediate feature vector set q1 and the intermediate feature vector set q2. In an embodiment of the present application, the fifth total number of first feature vectors in the intermediate feature vector set q1 is exemplified as 30, such as feature vector 1, feature vector 2,…, feature vector 30; and the fifth total number of first feature vectors in the intermediate feature vector set q2 is exemplified as 40, such as feature vector 31, feature vector 32,…, feature vector 70.

[0289] The business server compares the fifth total number of the first feature vectors in the intermediate feature vector set q1 with the second number threshold, and generates a leaf node for representing the intermediate feature vector set q1 because the fifth total number, which is 30, is less than the second number threshold, which is 50. The business server compares the fifth total number of the first feature vectors in the intermediate feature vector set q2 with the second number threshold, and generates a leaf node for representing the intermediate feature vector set q2 because the fifth total number, which is 40, is less than the second number threshold, which is 50.

[0290] According to the above processing procedure, the business server can obtain a binary tree corresponding to 1000 first feature vectors. The embodiment of the present application exemplifies the number of leaf nodes of the binary tree as 30. The feature vector sets corresponding to the 30 leaf nodes respectively include 1000 first feature vectors, and the feature vectors included in the 30 feature vector sets are different from each other.

[0291] The business server determines the first leaf node containing the sample feature vector from the above 30 leaf nodes. In the embodiment of the present application, the leaf node used to characterize the intermediate feature vector set q1 is exemplified as the first leaf node containing the sample feature vector.

[0292] The business server obtains a second leaf node adjacent to the leaf node used to characterize the intermediate feature vector set q1 from the above 30 leaf nodes. In the embodiment of the present application, the leaf node used to characterize the intermediate feature vector set q2 is exemplified as the second leaf node. The business server determines the intermediate feature vector set q1 and the intermediate feature vector set q2 as candidate feature vector sets, and the candidate feature vector set includes 70 first feature vectors.

[0293] The service server sends the sample feature vector (i.e. Figure 7 The feature vector 71c) in the candidate feature set is processed with the 70 feature vectors in the candidate feature set to obtain 70 candidate similarity values. The embodiment of the present application does not limit the specific method of similarity processing, and can be set according to actual conditions. The business server obtains the maximum candidate similarity value from the 70 candidate similarity values ​​and determines the maximum candidate similarity value as the first similarity value.

[0294] If the first total number of the first feature vectors in the first feature library is less than or equal to the first number threshold, see above Figure 2 as well as Figure 6 The descriptions in the respective corresponding embodiments.

[0295] Step S404: Compare the first similarity value with at least two candidate similarity thresholds corresponding to the first object, and obtain image prediction results corresponding to the at least two candidate similarity thresholds.

[0296] Step S405 , verifying at least two image prediction results using the label carried by the first sample image, and obtaining image prediction verification results corresponding to the at least two image prediction results.

[0297] Step S406: Determine a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds corresponding to the first object according to the at least two image prediction verification results.

[0298] Step S407, obtaining a second image containing a second object, performing feature extraction on the second image through an image recognition model to obtain a second feature vector; the second object is different from the first object.

[0299] Step S408, the second feature vector and the second label for indicating the second object are associated and stored in the first feature library to obtain a second feature library; the second feature library is used to determine together with the image recognition model that the first image to be detected includes the target object; the target object includes at least one of the first object or the second object.

[0300] For the specific implementation process of steps S404 to S408, please refer to the above Figure 6Steps S203 to S205 in the corresponding embodiment are not described in detail here.

[0301] As can be seen from the above, by constructing a feature library, the embodiment of the present application does not need to retrain the model when the number of objects to be identified increases, and the existing model can be used for reasoning, so the model training cost can be reduced. Since the feature vectors and labels in the feature library can be continuously enriched, the generalization of the image recognition model can be improved. In addition, when the number of first feature vectors in the first feature library is greater than the first quantity threshold, the embodiment of the present application constructs a binary tree, uses an indexing algorithm to find the first leaf node containing the sample feature vector, and determines the candidate feature vector set based on the first leaf node without performing a global search, so the indexing cost can be reduced. Since the structure of the binary tree can be continuously extended, the efficiency of the indexing algorithm can be improved.

[0302] For further information, see Fig.12 , Fig.12 1 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The above-mentioned data processing device 1 can be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.12 As shown, the data processing device 1 may include: an acquisition module 11 , an extraction module 12 and a storage module 13 .

[0303] The acquisition module 11 is used to acquire a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label stored in association; the first feature vector is used to represent a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image through the image recognition model;

[0304] An extraction module 12 is used to obtain a second image containing a second object, and extract features from the second image using an image recognition model to obtain a second feature vector; the second object is different from the first object;

[0305] The storage module 13 is used to associate and store the second feature vector and the second label indicating the second object in the first feature library to obtain a second feature library; the second feature library is used to jointly determine with the image recognition model that the first image to be detected includes a target object; the target object includes at least one of the first object or the second object.

[0306] The specific description of the data processing device 1 can be found in the above-mentioned data processing method, which will not be repeated here.

[0307] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0308] As can be seen from the above, when a second object different from the first object that has been added is added, the embodiment of the present application can extract features from the second image containing the second object through the image recognition model to obtain a second feature vector for characterizing the second image, add the second label for indicating the second object and the second feature vector to the first feature library to obtain the second feature library, and can determine that the image to be detected contains at least one of the first object or the second object through the second feature library and the image recognition model. Obviously, by constructing a feature library, the embodiment of the present application does not need to retrain the model when the number of objects to be recognized increases, and can use the existing model for reasoning, so the model training cost can be reduced, and since the feature vectors and labels in the feature library can be continuously enriched, the generalization of the image recognition model can be improved.

[0309] For further information, see Fig.13 , Fig.13 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be Figure 1 The terminal device or service server shown in FIG. Fig.13 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize connection and communication between these components.

[0310] In some embodiments, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1005 may also be at least one storage device located away from the aforementioned processor 1001.

[0311] like Fig.13As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0312] exist Fig.13 In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to implement the method steps in each embodiment of the present application.

[0313] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the description of the data processing method or device in the above embodiments, which will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated.

[0314] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the description of the data processing method or device in the above embodiments is implemented, which will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated.

[0315] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., provided on the computer device.

[0316] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0317] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device can execute the description of the data processing method or device in the above embodiments, which will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated.

[0318] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0319] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. A data processing method, characterized in that: include: Acquire a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label stored in association; the first feature vector is used to represent a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image using the image recognition model; Acquire a second image containing a second object, and extract features from the second image using the image recognition model to obtain a second feature vector; the second object is different from the first object; The second feature vector and the second label indicating the second object are associated and stored in the first feature library to obtain a second feature library; the second feature library is used to determine together with the image recognition model that the first image to be detected includes a target object; the target object includes at least one of the first object or the second object.

2. The method according to claim 1, characterized in that Also includes: Obtaining a first sample image in a test set, performing feature extraction on the first sample image using the image recognition model, and obtaining a sample feature vector corresponding to the first sample image; According to the sample feature vector and the first feature vector, a target similarity threshold corresponding to the first object is determined from at least two candidate similarity thresholds corresponding to the first object; the target similarity threshold is used together with the first feature vector and the image recognition model to jointly determine that a second image to be detected includes the first object.

3. The method according to claim 2, characterized in that The step of determining, based on the sample feature vector and the first feature vector, a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds corresponding to the first object comprises: Performing similarity processing on the sample feature vector and the first feature vector to obtain a first similarity value; Comparing the first similarity value with at least two candidate similarity thresholds corresponding to the first object, respectively, to obtain image prediction results corresponding to the at least two candidate similarity thresholds; Verifying at least two image prediction results by using the label carried by the first sample image to obtain image prediction verification results corresponding to the at least two image prediction results respectively; According to at least two image prediction verification results, a target similarity threshold corresponding to the first object is determined from at least two candidate similarity thresholds corresponding to the first object.

4. The method according to claim 3, characterized in that The at least two candidate similarity thresholds corresponding to the first object include candidate similarity thresholds C b , the at least two image prediction results include the candidate similarity threshold C b The corresponding image prediction result A b , b is a positive integer, and b is less than or equal to the total number of the at least two candidate similarity thresholds; The verifying process of at least two image prediction results by using the label carried by the first sample image to obtain image prediction verification results respectively corresponding to the at least two image prediction results includes: If the label carried by the first sample image is the first label for indicating that the first sample image contains the first object, then when the first similarity value is greater than or equal to the candidate similarity threshold C b When the image prediction result is A, the correct image prediction result is determined as the image prediction result A b Image prediction verification results; If the label carried by the first sample image is an invalid label for indicating that the first sample image does not contain the first object, then when the first similarity value is less than the candidate similarity threshold C b When the image prediction result is A, the correct image prediction result is determined as the image prediction result A b Image prediction verification results; If the label carried by the first sample image is the first label, then when the first similarity value is less than the candidate similarity threshold C b When the image prediction error result is determined as the image prediction result A b Image prediction verification results; If the label carried by the first sample image is the invalid label, then when the first similarity value is greater than or equal to the candidate similarity threshold C b When the image prediction error result is determined as the image prediction result A b Image prediction verification results.

5. The method according to claim 4, characterized in that The step of comparing the first similarity value with at least two candidate similarity thresholds corresponding to the first object to obtain image prediction results corresponding to the at least two candidate similarity thresholds respectively includes: The first similarity value is compared with the candidate similarity threshold C b If the first similarity value is greater than or equal to the candidate similarity threshold C b , then the positive result of image prediction is determined as the candidate similarity threshold C b a corresponding image prediction result; the positive image prediction result is used to predict that the first sample image contains the first object; If the first similarity value is less than the candidate similarity threshold C b , then the negative result of image prediction is determined as the candidate similarity threshold C b The corresponding image prediction result; the image prediction negative result is used to predict that the first sample image does not contain the first object.

6. The method according to claim 3, characterized in that The at least two candidate similarity thresholds corresponding to the first object include candidate similarity thresholds C b The at least two image prediction verification results include the candidate similarity threshold C b Corresponding image prediction verification result D b , b is a positive integer, and b is less than or equal to the total number of the at least two candidate similarity thresholds; The step of determining, based on at least two image prediction verification results, a target similarity threshold corresponding to the first object from at least two candidate similarity thresholds corresponding to the first object comprises: According to the image prediction verification result D b , determine the candidate similarity threshold C b a first accurate value for the first sample image; Get the candidate similarity threshold C b a second accurate value for a second sample image; the second sample image refers to the remaining sample images in the test set except the first sample image; Determine the candidate similarity threshold C according to the first accurate value, the second accurate value and the total number of all sample images in the test set b a corresponding first evaluation value; According to the first evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds, a target similarity threshold corresponding to the first object is determined from the at least two candidate similarity thresholds.

7. The method according to claim 6, characterized in that The candidate similarity threshold C is determined according to the first accurate value, the second accurate value and the total number of all sample images in the test set. b The corresponding first evaluation value includes: If the label carried by the first sample image is the first label, then the verification result D is predicted according to the image b , determine the candidate similarity threshold C b a first recall value for the first sample image; Get the candidate similarity threshold C b a second recall value for a third sample image; the third sample image refers to a sample image in the test set that carries the first label except the first sample image; Determine the candidate similarity threshold C according to the first accurate value, the second accurate value and the total number of all sample images in the test set b The corresponding total accurate value; Determine the candidate similarity threshold C according to the first recall value, the second recall value, and the total number of sample images carrying the first label in the test set. b The corresponding total recall value; The total accuracy value and the total recall value are weighted and summed to obtain the candidate similarity threshold C. b The corresponding first evaluation value.

8. The method according to claim 6, characterized in that The determining, according to the first evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds, a target similarity threshold corresponding to the first object from the at least two candidate similarity thresholds includes: Obtaining a maximum first evaluation value from the first evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds; The candidate similarity threshold corresponding to the maximum first evaluation value among the at least two candidate similarity thresholds is determined as the target similarity threshold corresponding to the first object.

9. The method according to claim 6, characterized in that Also includes: Obtain a sample video, perform frame extraction processing on the sample video, and obtain E sample images belonging to the sample video, where E is a positive integer; The E sample images include a first sample image; Add the E sample images to the test set; Then, determining the target similarity threshold corresponding to the first object from the at least two candidate similarity thresholds according to the first evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds includes: Get the E sample images for the candidate similarity threshold C b The corresponding image prediction verification results; The number of results corresponding to the correct image prediction results in the E image prediction verification results is compared with the candidate similarity threshold C b The corresponding F candidate frame thresholds are compared respectively to obtain the video prediction results corresponding to the F candidate frame thresholds respectively; F is a positive integer greater than 1; By using the label carried by the sample video, the F video prediction results are respectively verified to obtain video prediction verification results corresponding to the F video prediction results; the label carried by the sample video is the first label used to indicate that the sample video contains the first object, or an invalid label used to indicate that the sample video does not contain the first object; Determine the second evaluation values ​​corresponding to the F candidate frame thresholds respectively according to the F video prediction verification results; wherein one video prediction verification result is used to determine the second evaluation value corresponding to one candidate frame threshold; According to the F second evaluation values ​​and the candidate similarity threshold C b A corresponding first evaluation value is used to determine a target similarity threshold corresponding to the first object from the at least two candidate similarity thresholds.

10. The method according to claim 9, characterized in that The F candidate frame thresholds include a candidate frame threshold G h , h is a positive integer, and h is less than or equal to F; the F second evaluation values ​​include the candidate frame threshold G h a corresponding second evaluation value; The method according to the F second evaluation values ​​and the candidate similarity threshold C b The method further comprises: determining a target similarity threshold value corresponding to the first object from the at least two candidate similarity threshold values, and comprising: The candidate frame threshold G h The corresponding second evaluation value and the candidate similarity threshold C b The corresponding first evaluation values ​​are weighted and summed to obtain the candidate similarity threshold C b And the candidate frame threshold G h The total evaluation value of the combined candidate parameter pairs; The candidate similarity threshold C b The total evaluation values ​​corresponding to the F candidate parameter pairs are determined as the candidate similarity threshold C b The corresponding F total evaluation values; Obtaining F total evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds, and obtaining a maximum total evaluation value among the F total evaluation values ​​respectively corresponding to the at least two candidate similarity thresholds; Determine the candidate similarity threshold corresponding to the maximum total evaluation value among the at least two candidate similarity thresholds as the target similarity threshold corresponding to the first object; The candidate frame threshold corresponding to the maximum total evaluation value among the at least two candidate frame thresholds is determined as the target frame threshold corresponding to the first object; the target parameter pair composed of the target similarity threshold and the target frame threshold is used together with the first feature vector and the image recognition model to jointly determine that the second image to be detected includes the first object.

11. The method according to claim 3, characterized in that The first total number of the first feature vectors is I; I is a positive integer greater than 1; The performing similarity processing on the sample feature vector and the first feature vector to obtain a first similarity value includes: If I is greater than a first quantity threshold, the I first eigenvectors are divided to obtain a binary tree corresponding to the I first eigenvectors; the binary tree includes at least two leaf nodes; the eigenvector sets corresponding to the at least two leaf nodes respectively include the I first eigenvectors, and the eigenvectors respectively included in the at least two eigenvector sets are different from each other; Among the at least two leaf nodes, determining a first leaf node that includes the sample feature vector; According to the first leaf node, a candidate feature vector set is determined; the candidate feature vector set belongs to the I first feature vectors; the candidate feature vector set includes j feature vectors; j is a positive integer greater than 1; The sample feature vector and the j feature vectors are respectively processed for similarity to obtain j candidate similarity values, and a maximum candidate similarity value of the j candidate similarity values ​​is determined as a first similarity value.

12. The method according to claim 11, characterized in that The step of determining a set of candidate feature vectors according to the first leaf node includes: Among the at least two leaf nodes, obtain a second leaf node adjacent to the first leaf node; The feature vector set corresponding to the first leaf node and the feature vector set corresponding to the second leaf node are determined as candidate feature vector sets.

13. The method according to claim 11, characterized in that The dividing process of the I first eigenvectors to obtain a binary tree corresponding to the I first eigenvectors includes: Randomly select two feature vectors from I first feature vectors as a first segmentation vector pair, use a hyperplane between the first segmentation vector pair as a segmentation plane, and perform segmentation processing on the I first feature vectors to obtain two first intermediate feature vector sets; the two first intermediate feature vector sets include an intermediate feature vector set k, and the intermediate feature vector set k is any first intermediate feature vector set in the two first intermediate feature vector sets; Determine a second total number of feature vectors in the intermediate feature vector set k, and if the second total number is greater than a second number threshold, randomly select two feature vectors in the intermediate feature vector set k as a second segmentation vector pair; The hyperplane between the second segmentation vector pairs is used as a segmentation plane, and the intermediate feature vector set k is divided to obtain two second intermediate feature vector sets; the two second intermediate feature vector sets include an intermediate feature vector set m, and the intermediate feature vector set m is any second intermediate feature vector set in the two second intermediate feature vector sets; Determining a third total number of feature vectors in the intermediate feature vector set m, and if the mth total number is equal to or less than the second number threshold, generating a leaf node for representing the intermediate feature vector set m; According to the leaf nodes used to characterize the intermediate feature vector set m, a binary tree corresponding to the I first feature vectors is constructed.

14. The method according to claim 1, characterized in that The acquiring of a second image containing a second object comprises: Acquire a target video containing a second object, and perform frame extraction processing on the target video to obtain at least two candidate images; Acquire an image containing the second object from among the at least two candidate images, determine the acquired image as an initial image, and perform data enhancement processing on the initial image to obtain an enhanced image different from the initial image; The initial image and the enhanced image are determined as the second image.

15. The method according to claim 1, characterized in that Also includes: Determine the total number of first vectors of feature vectors in the second feature library; the feature vectors in the second feature library include the first feature vector and the second feature vector; If the total number of the first vectors is greater than the vector number threshold, then the feature vector N p Perform dimensionality reduction processing to obtain the feature vector N p The corresponding simplified eigenvector; The feature vector N p is any feature vector in the second feature library; The feature vector N p The vector dimension of is greater than the vector dimension of the simplified feature vector, and the simplified feature vector includes the feature vector N p key information.

16. The method according to claim 1, characterized in that Also includes: Determine the total number of first vectors of feature vectors in the second feature library; the feature vectors in the second feature library include the first feature vector and the second feature vector; the total number of second vectors of the first feature vector is at least two; If the total number of the first vectors is greater than the vector number threshold, at least two first feature vectors are fused to obtain a synthetic feature vector for characterizing the first object.

17. A data processing device, characterized in that: include: an acquisition module, configured to acquire a first feature library and an image recognition model; the first feature library includes a first feature vector and a first label stored in association; the first feature vector is used to represent a first image containing a first object; the first label is used to indicate the first object; the first feature vector is obtained by extracting features from the first image using the image recognition model; an extraction module, configured to obtain a second image containing a second object, and perform feature extraction on the second image using the image recognition model to obtain a second feature vector; the second object is different from the first object; A storage module is used to associate and store the second feature vector and the second label indicating the second object in the first feature library to obtain a second feature library; the second feature library is used to jointly determine with the image recognition model that the first image to be detected includes a target object; the target object includes at least one of the first object or the second object.

18. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1-16.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 16.

20. A computer program product, characterized in that The computer program product comprises a computer program, which is stored in a computer-readable storage medium. The computer program is suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 16.

Images