Model fine tuning method and device, equipment, storage medium and program product

Abstract

The invention discloses a model fine tuning method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring an image recognition result of recognizing video data by an image recognition model and a feedback label of a user on the image recognition result; performing key feature extraction on the video data based on the feedback tag to obtain a plurality of key local features; performing feature clustering on a plurality of first recognition features in the image recognition result based on the plurality of key local features to obtain a clustering result corresponding to each key local feature, the clustering result being used for representing a recognition feature cluster having the same feedback tag as the key local feature; based on the clustering result and the corresponding feedback label, constructing a model fine tuning sample; and performing fine tuning training on the image recognition model according to the model fine tuning sample to obtain a fine-tuned image recognition model. According to the embodiment of the invention, the optimization effect can be improved, and the recognition accuracy of the image recognition model is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of image recognition, and in particular, relates to a model fine-tuning method, device, equipment, storage medium and program product. Background Art

[0002] In the field of image recognition, neural network models are usually used to quickly identify images, and the accuracy of neural network model recognition depends on the training method of the neural network model, especially the optimization of model parameters in the neural network model during the training process.

[0003] At present, the model parameters of the neural network model are generally optimized by fine-tuning, prompt fine-tuning, adapter fine-tuning, parameter fine-tuning and remapping fine-tuning. In the process of optimizing the neural network model by the above optimization methods, the changes in the global structural parameters of the neural network model during the optimization process are usually considered, and the global structural parameters of the neural network model are adjusted. However, the neural network model optimized by the above optimization methods will have problems such as poor optimization effect and inaccurate recognition effect in specific practical application scenarios. Summary of the invention

[0004] The embodiments of the present application provide a model fine-tuning method, apparatus, device, storage medium and program product, which can improve the optimization effect and improve the accuracy of neural network model recognition.

[0005] On the one hand, an embodiment of the present application provides a model fine-tuning method, the method comprising:

[0006] Obtaining an image recognition result of an image recognition model identifying video data and a user's feedback label on the image recognition result, wherein the image recognition result includes a plurality of first recognition features;

[0007] Extract key features from the video data based on the feedback tags to obtain multiple key local features;

[0008] Performing feature clustering on the multiple first recognition features in the image recognition result based on the multiple key local features to obtain a clustering result corresponding to each of the key local features, wherein the clustering result is used to characterize a recognition feature cluster having the same feedback label as the key local feature;

[0009] Based on the clustering results and the corresponding feedback labels, construct a model fine-tuning sample;

[0010] The image recognition model is fine-tuned and trained according to the model fine-tuning samples to obtain a fine-tuned image recognition model.

[0011] Optionally, extracting key features from the video data based on the feedback tag to obtain a plurality of key local features includes:

[0012] Extracting video frame images from the video data to obtain multiple video frame images;

[0013] Constructing a scale space of each of the video frame images, the scale space including video frame images at multiple scales, the scales being used to represent different clarity levels of the video frame images;

[0014] For each video frame image at the scale, determine a plurality of pixel extreme value points of the video frame image, wherein the pixel extreme value points represent pixel points corresponding to the maximum pixel response value or the minimum pixel response value at the scale, and the pixel response value reflects the stability and importance of the pixel points;

[0015] Determine the feature descriptor corresponding to each of the pixel extreme points respectively, wherein the feature descriptor is used to represent the local feature vector of the video frame image;

[0016] The feature descriptors are spliced ​​based on a preset row direction to obtain a plurality of key local features of the video frame image at each scale.

[0017] Optionally, the determining a plurality of pixel extreme points of the video frame image includes:

[0018] Determine the local curvature of each pixel point based on a preset curvature calculation formula;

[0019] For each of the pixel points, determining a pixel response value of the pixel point based on a local curvature corresponding to the pixel point and a preset constant;

[0020] A pixel extreme point is determined from the pixel points corresponding to the plurality of pixel response values.

[0021] Optionally, before determining the pixel extreme point from the pixel points corresponding to the plurality of pixel response values, the method further comprises:

[0022] Based on the local curvature and the pixel response value, performing sub-pixel precision processing on each pixel point to obtain a sub-pixel offset vector;

[0023] Determining a key pixel point based on the sub-pixel offset vector;

[0024] Determining a pixel extreme point from the pixel points corresponding to the plurality of pixel response values ​​comprises:

[0025] A pixel extreme point is determined from the key pixel points corresponding to the plurality of pixel response values.

[0026] Optionally, respectively determining the feature descriptor corresponding to each of the pixel extreme points includes:

[0027] Establish an image coordinate system based on the video frame image at each scale;

[0028] For each of the pixel extreme value points, obtaining the pixel coordinates corresponding to the pixel extreme value point from the image coordinate system;

[0029] Based on the pixel coordinates and a first preset local range, determining a local pixel area, wherein the local pixel area includes a plurality of pixel points;

[0030] In the local pixel area, based on a preset gradient algorithm, determining a first gradient value in the horizontal direction and a second gradient value in the vertical direction corresponding to each pixel point in the local pixel area;

[0031] Determine the amplitude and direction corresponding to the pixel extreme point based on the first gradient value and the second gradient value;

[0032] A feature descriptor of the local pixel area is generated based on the amplitude and direction corresponding to the pixel extreme point.

[0033] Optionally, determining a feature descriptor of each local pixel area based on the amplitude and direction corresponding to the pixel point includes:

[0034] Dividing the local pixel area into blocks to obtain a plurality of pixel blocks;

[0035] For each of the pixel blocks, counting direction histograms corresponding to a plurality of the pixel points in the pixel block, the direction histogram reflecting gradient direction distribution information about the pixel extreme value points in the pixel block;

[0036] The distribution histogram corresponding to each pixel block is counted to obtain a feature descriptor of each local pixel area.

[0037] Optionally, the clustering result includes at least one target cluster, and the performing feature clustering on the multiple first recognition features in the image recognition result based on the multiple key local features to obtain the clustering result corresponding to each of the key local features includes:

[0038] Based on the plurality of key local features and the preset local radius, a plurality of second local ranges are determined, wherein the second local ranges include a key local feature and at least one second identification feature, and the second identification feature includes identification features of the plurality of first identification features except the key local feature;

[0039] Determine, within the second local range, a first local density and a first average distance of a target local feature, wherein the target local feature includes a key local feature or a second local feature;

[0040] Determining a clustering score of the target local feature based on the first local density and the first average distance;

[0041] Based on the clustering score, determining at least one initial clustering core;

[0042] Based on the initial clustering kernel and a preset distance threshold, the key local features within the second local range are clustered to obtain at least one target cluster.

[0043] Optionally, clustering the key local features within the second local range based on the initial clustering kernel and a preset distance threshold to obtain at least one target cluster includes:

[0044] Determine a first distance between each of the key local features and the initial clustering core respectively, the initial clustering core being the initial clustering core that is closest to the key local features;

[0045] Based on the first distance and the preset distance threshold, an initial cluster is determined, and the initial cluster is determined as the target cluster.

[0046] Optionally, after determining the initial cluster based on the first distance and the preset distance threshold, the method further includes:

[0047] Counting the number of key local features of the initial clustering clusters;

[0048] Determine a first distance mean of each of the initial clustering clusters based on the key local feature vector and the first distance;

[0049] Based on the first distance mean, determining an updated clustering core of the initial clustering cluster;

[0050] In the case that the updated clustering core does not meet the preset stop updating condition, repeating the step of respectively determining the first distance between each of the key local features and the initial clustering core, wherein the initial clustering core is the initial clustering core with the closest distance to the key local features, until the updated clustering core meets the preset stop updating condition, and determining the updated clustering core as the target clustering core;

[0051] Based on the target clustering core, the initial clustering cluster is updated to obtain a target clustering cluster.

[0052] Optionally, the image recognition result includes a first recognition result and a second recognition result, and the feedback tag corresponds to the first recognition result;

[0053] The feedback tag includes at least one incorrectly identified target feature and / or at least one correctly identified target feature; the clustering result includes a first cluster corresponding to the incorrectly identified target feature and a second cluster corresponding to the correctly identified target feature;

[0054] The performing feature clustering on the plurality of first recognition features in the image recognition result based on the plurality of key local features to obtain a clustering result corresponding to each of the key local features further includes:

[0055] Acquire a plurality of second identification features in the second identification result;

[0056] For each second identification feature:

[0057] Calculating a first similarity between the second identification feature and the first cluster;

[0058] Calculating a second similarity between the second identification feature and the second cluster;

[0059] When the first similarity is smaller than a first preset similarity and the second similarity is larger than a second preset similarity, the second identification feature corresponding to the first similarity is classified into the second cluster.

[0060] On the other hand, an embodiment of the present application provides a model fine-tuning device, the device comprising:

[0061] An acquisition module, used to acquire an image recognition result of an image recognition model identifying video data and a user's feedback label on the image recognition result, wherein the image recognition result includes a plurality of first recognition features;

[0062] A feature extraction module, used to extract key features of the video data based on the feedback tags to obtain multiple key local features;

[0063] A feature clustering module, used for performing feature clustering on the multiple first recognition features in the image recognition result based on the multiple key local features to obtain a clustering result corresponding to each of the key local features, wherein the clustering result is used to characterize a recognition feature cluster having the same feedback label as the key local feature;

[0064] A construction module, used to construct a model fine-tuning sample based on the clustering results and the corresponding feedback labels;

[0065] The fine-tuning training module is used to perform fine-tuning training on the image recognition model according to the model fine-tuning sample to obtain a trained image recognition model.

[0066] In another aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0067] When the processor executes the computer program instructions, the model fine-tuning method as described in the first aspect is implemented.

[0068] On the other hand, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the model fine-tuning method described in the first aspect is implemented.

[0069] On the other hand, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the model fine-tuning method as described in the first aspect.

[0070] The model fine-tuning method, apparatus, device and computer storage medium of the embodiments of the present application can, when performing model fine-tuning training, extract key point features of video data based on user feedback labels of image recognition results to obtain multiple key local features. These key local features integrate user feedback and can represent the true recognition accuracy of the image recognition model. Then, the first recognition features are clustered through these key local features, and the first recognition features can be adaptively fused. At the same time, the key local features can take into account the overall structure of multiple first recognition features of video data on the recognition influence of the image recognition model, thereby improving the fine-tuning training effect of the image recognition model and thus improving the recognition accuracy of the image recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0072] Figure 1 It is a flowchart of the model fine-tuning method provided in the first embodiment of the present application;

[0073] Figure 2 is a flow chart of a model fine-tuning method provided in the second embodiment of the present application;

[0074] Figure 3 is a flow chart of a model fine-tuning method provided in the third embodiment of the present application;

[0075] Figure 4 is a flowchart of a model fine-tuning method provided in the fourth embodiment of the present application;

[0076] Figure 5 is a flowchart of a model fine-tuning method provided in the fifth embodiment of the present application;

[0077] Figure 6 is a flowchart of a model fine-tuning method provided in the sixth embodiment of the present application;

[0078] Figure 7 is a flowchart of a model fine-tuning method provided in the seventh embodiment of the present application;

[0079] Figure 8 is a structural schematic diagram of a model fine-tuning device provided in another embodiment of the present application;

[0080] Fig. 9 It is a structural schematic diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0081] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0082] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0083] In order to solve the problems of the prior art, the embodiments of the present application provide a model fine-tuning method, device, equipment, storage medium and program product. When performing model fine-tuning training, it is possible to extract key point features of video data based on user feedback labels of image recognition results to obtain multiple key local features, which integrate user feedback and can represent the true recognition accuracy of the image recognition model. Then, the first recognition features are clustered through these key local features, and the first recognition features can be adaptively fused. At the same time, the overall structure of multiple first recognition features of video data can be considered through the key local features. The recognition influence of the image recognition model, improve the fine-tuning training effect of the image recognition model, and thus improve the recognition accuracy of the image recognition model.

[0084] The following first introduces the model fine-tuning method provided in the embodiment of the present application.

[0085] Figure 1 FIG. 1 is a flow chart of a model fine-tuning method provided by an embodiment of the present application. Figure 1 As shown, the model fine-tuning method may specifically include S101-S105:

[0086] S101, obtaining an image recognition result of an image recognition model identifying video data and a user's feedback label on the image recognition result.

[0087] In some embodiments, in S101, as an example, the image recognition result includes multiple first recognition features; when fine-tuning the image recognition model, the image recognition model is first required to recognize the video data to obtain a preliminary image recognition result. The image recognition result is the recognition result of the image recognition model that has not been fine-tuned and trained, and there may be recognition errors. At this time, the image recognition result can be sent to the user, and then through the feedback label of the user's feedback on the image recognition result, it can be understood which first recognition features in the image recognition result are recognized incorrectly and which are recognized correctly, that is, by marking the first recognition feature that is recognized incorrectly with a feedback label of the incorrectly recognized target feature, and marking the first recognition feature that is recognized correctly with a feedback label of the correctly recognized target feature.

[0088] S102, extracting key features of the video data based on the feedback tags to obtain multiple key local features.

[0089] In some embodiments, in S102, when extracting key features of video data, in order to better cluster the training data to achieve accurate fine-tuning of the image recognition model, before clustering, it is necessary to extract key features of the video data based on the feedback tags of user feedback to obtain key local features associated with the feedback tags, and then construct target features for clustering through these key local features to complete the clustering of multiple first recognition features in the video data.

[0090] In this embodiment, the key local features can be extracted by constructing a scale space, which can extract the key local features more accurately.

[0091] S103, performing feature clustering on the multiple first recognition features in the image recognition result based on the multiple key local features to obtain a clustering result corresponding to each key local feature.

[0092] In some embodiments, in S103 , as an example, the clustering result is used to characterize the recognition feature clusters having the same feedback label as the key local feature.

[0093] In this embodiment, since each video data includes a large number of video images and each video image involves multiple first recognition features, in order to better fine-tune the image recognition model, it is necessary to cluster the multiple first recognition features, that is, to divide different recognition feature clusters according to each key local feature, and each recognition feature cluster represents different correct recognition features and incorrect recognition features.

[0094] As an example, the algorithm for performing feature clustering may be a k-means clustering algorithm or other clustering algorithms, as long as the clustering of the first identification feature can be achieved, and no further details will be given here.

[0095] In this embodiment, clustering the multiple first recognition features can make the clustering result more robust, and the obtained recognition feature clusters are easier to converge to the global optimum, which is convenient for fine-tuning the image recognition model.

[0096] S104: construct a model fine-tuning sample based on the clustering results and the corresponding feedback labels.

[0097] In some embodiments, in S104, after obtaining the clustering results, it is also necessary to associate the clustering results with the feedback labels to obtain model fine-tuning samples that are consistent with user feedback, so as to use the model training samples to fine-tune the image recognition model more accurately, thereby making the trained image recognition model more accurate in recognizing video data.

[0098] As an example, a model fine-tuning sample may include a recognition feature cluster corresponding to a feedback label and its corresponding feedback label, and then construct a model fine-tuning sample by respectively constructing the recognition feature clusters corresponding to all correctly recognized target features and incorrectly recognized target features identified by the user to train the image recognition model.

[0099] S105, fine-tuning the image recognition model according to the model fine-tuning sample to obtain a fine-tuned image recognition model.

[0100] In some embodiments, in S105, in the process of fine-tuning the image recognition model using the model fine-tuning samples, the image recognition model is continuously iteratively trained using the model fine-tuning samples.

[0101] It is worth noting that in this embodiment, during the iterative training process, after a period of training, user feedback can be provided on the image recognition results of the image recognition model to optimize the recognition accuracy of the trained image recognition model. Repeated tuning can be performed continuously to continuously update the model fine-tuning samples, thereby continuously improving the accuracy of the image recognition model.

[0102] In an embodiment of the present application, when fine-tuning the model, key point features of the video data can be extracted based on the user's feedback labels on the image recognition results to obtain multiple key local features. These key local features integrate the user's feedback and can represent the true recognition accuracy of the image recognition model. Then, the first recognition features are clustered through these key local features, and the first recognition features can be adaptively fused. At the same time, the key local features can be used to consider the impact of the overall structure of multiple first recognition features of the video data on the recognition of the image recognition model, thereby improving the fine-tuning training effect of the image recognition model and thus improving the recognition accuracy of the image recognition model.

[0103] In some other embodiments, in S101, in order to obtain a more accurate feedback label, the video data can be first subjected to frame extraction processing, that is, video images are extracted for each frame of the video data to obtain multiple video frame images, and then the video frame images are input into an image recognition model. The image recognition model recognizes the multiple video frame images to obtain image recognition results, and the image recognition results include multiple first recognition features, that is, each first recognition feature is a recognition target of the video frame image.

[0104] Afterwards, the user reviews part or all of the image recognition results, marks multiple incorrect and correct first recognition features, and forms feedback labels. For example, the recognition target is a car, but some video frame images are recognized as cars, while some video frame images are recognized as obstacles. At this time, the image recognition result that recognizes the car can be marked with the first feedback label of correct recognition of the target feature, and the image recognition result that recognizes the obstacle can be marked with the second feedback label of incorrect recognition of the target feature.

[0105] At this time, the first feedback label and the second feedback label fed back by the user and the corresponding video frame image can be used as original samples to construct a model fine-tuning sample.

[0106] It is worth noting that, in this embodiment, the user's feedback on a part of the image recognition results is taken as an example for explanation.

[0107] Reference Figure 2 In some other embodiments, in order to obtain more accurate key local features, S102 may include S1021-S1025:

[0108] S1021, extracting video frame images from the video data to obtain a plurality of video frame images;

[0109] S1022, constructing a scale space of each video frame image;

[0110] S1023, determining a plurality of pixel extreme value points of the video frame image at each scale;

[0111] S1024, respectively determining the feature descriptor corresponding to each pixel extreme point;

[0112] S1025, splicing feature descriptors based on a preset row direction to obtain multiple key local features of the video frame image at each scale.

[0113] In some embodiments, S1021 is consistent with the frame extraction process in S101 above to obtain the video frame image, which will not be repeated here.

[0114] In some other embodiments, in S1022, this embodiment adopts key feature extraction based on scale space, so it is first necessary to construct a scale space based on the video frame image corresponding to the image recognition result obtained in S101. In this scale space, the scale space may include video frame images at multiple scales, and the scale can be used to represent different clarity levels of the video frame image, that is, the grayscale value is adjusted for each video frame image to obtain a scale space with different grayscale clarity.

[0115] It is worth noting that, in this embodiment, in each scale space, the scale of the pixel can be reflected by the gray value of the pixel.

[0116] As an example, a Gaussian filter may be used to blur the video frame image to obtain the video frame image at each different grayscale value.

[0117] In some embodiments, in S1023, as an example, the pixel extreme point represents the pixel point corresponding to the maximum pixel response value or the pixel point corresponding to the minimum pixel response value under the scale, and the pixel response value reflects the stability and importance of the pixel point.

[0118] In this embodiment, in order to obtain more accurate key local features, multiple pixel extreme points, that is, the most stable and important pixel points, can be determined from different clarity for the video frame image at each scale, and then these pixel points can be used as key pixel points for target recognition features. These key pixel points can depict the target recognition features, so that the subsequent fine-tuning training of the image recognition model will be more accurate.

[0119] Reference Figure 3 In some other embodiments, S1023 may specifically include S301-S303:

[0120] S301, determining the local curvature of each pixel point based on a preset curvature calculation formula;

[0121] S302, for each pixel point, determining a pixel response value of the pixel point based on a local curvature corresponding to the pixel point and a preset constant;

[0122] S303, determining a pixel extreme point from pixel points corresponding to a plurality of pixel response values.

[0123] In this embodiment, in S301, in the process of determining the pixel extreme points, the Hessian Matrix of each video frame image can be calculated first, and the corner points and edges of the video frame image are detected by the Hessian Matrix, that is, the local curvature of each pixel point in each video frame image is obtained.

[0124] Specifically, the local curvature can be determined by the following formula (1):

[0125] H(x,y,σ)=[L xx (x,y,σ)L xy (x,y,σ)L xy (x,y,σ)L yy (x,y,σ)] (1)

[0126] Among them, H(x, y, σ) represents the local curvature of the pixel point (x, y), Lxx (x,y,σ),L xy (x,y,σ),L xy (x,y,σ),L yy (x, y, σ) represents the second-order derivative at each scale.

[0127] In this embodiment, in S302, for each pixel point, a pixel response value of each pixel point needs to be determined to determine a pixel extreme value point.

[0128] Specifically, the pixel response value can be determined by the following formula (2):

[0129]

[0130] Among them, R(x,y,σ) represents the pixel response value of the pixel point (x,y), trace(H(x,y,σ)) represents the trace of the Hessian matrix, det(H(x,y,σ)) represents the determinant of the Hessian matrix, and ∈ is a preset constant, which is a positive integer.

[0131] In this embodiment, after determining each pixel response value, it is necessary to determine the pixel extreme point according to the pixel response value corresponding to each pixel point, that is, in S303, from the pixel points corresponding to multiple pixel response values, the pixel point corresponding to the maximum pixel response value or the minimum pixel response value is taken as the pixel extreme point.

[0132] Specifically, the pixel response values ​​of the current pixel point and the pixel points adjacent to the current pixel point can be compared one by one to determine the maximum pixel response value or the minimum pixel response value from multiple pixel response values, and then the pixel point corresponding to the maximum pixel response value or the minimum pixel response value is determined as the pixel extreme point.

[0133] In this embodiment, by determining the pixel extreme value points of each target recognition feature in each video frame image, the key local features corresponding to each target recognition feature can be better determined, so as to achieve better clustering of the first recognition features.

[0134] Reference Figure 4 In some other embodiments, in order to improve the accuracy of extracting key local features, before S303, S401-S403 may also be included:

[0135] S401, performing sub-pixel precision processing on each pixel point based on the local curvature and the pixel response value to obtain a sub-pixel offset vector;

[0136] S402, determining key pixel points based on the sub-pixel offset vector;

[0137] S303 may include:

[0138] S403, determining a pixel extreme point from key pixel points corresponding to a plurality of pixel response values.

[0139] In this embodiment, sub-pixel precision processing can be performed based on the local curvature and pixel response value calculated above to obtain a sub-pixel offset vector corresponding to each pixel point, and then the pixel point is refined through the sub-pixel offset vector to obtain the key pixel point corresponding to each pixel point.

[0140] Specifically, the sub-pixel offset vector can be determined by the following formula (3):

[0141]

[0142] Where D is the sub-pixel offset vector, H -1 is the inverse of the Hessian matrix of the pixel point, The gradient corresponding to the pixel response value.

[0143] In this embodiment, by performing sub-pixel precise processing on pixels, the accuracy and resolution of pixels can be improved, and a more refined division of the grayscale value of each pixel can be achieved, so as to improve more accurate processing of video frame images.

[0144] In some other embodiments, in S1024, the feature descriptor is used to represent the local feature vector of the video frame image, that is, in order to determine the target recognition features more quickly and accurately, the feature descriptor is calculated in the surrounding area of ​​each pixel extreme point to construct the key local features.

[0145] Reference Figure 5 Specifically, S1024 may include S501-S506:

[0146] S501, establishing an image coordinate system based on the video frame image at each scale;

[0147] S502, for each pixel extreme point, obtaining the pixel coordinates corresponding to the pixel extreme point from the image coordinate system;

[0148] S503, determining a local pixel area based on the pixel coordinates and the first preset local range;

[0149] S504, determining, in the local pixel area, a first gradient value in the horizontal direction and a second gradient value in the vertical direction corresponding to each pixel point in the local pixel area based on a preset gradient algorithm;

[0150] S505, determining the amplitude and direction corresponding to the pixel extreme point based on the first gradient value and the second gradient value;

[0151] S506, generating a feature descriptor of the local pixel area based on the amplitude and direction corresponding to the pixel extreme point.

[0152] In this embodiment, in the process of determining the feature descriptor, an image coordinate system is first established based on the video frame image, and then the pixel coordinates corresponding to each pixel extreme point are obtained. Thereafter, with each pixel extreme point as the center and the first preset local range as the radius, the local pixel area is determined, and the local pixel area includes multiple pixel points, that is, multiple pixels are contained in the local pixel area.

[0153] For each local pixel area, the gradient of the video frame image is calculated. That is, in S504, the first gradient value in the horizontal direction can be determined by the following formula (4) and the second gradient value in the vertical direction can be determined by formula (5):

[0154]

[0155]

[0156] Among them, G x Represents the gradient of the video frame image in the x direction, G y Represents the gradient of the video frame image in the y direction.

[0157] Then, the amplitude and direction of each pixel are determined by the first gradient value and the second gradient value to generate a feature descriptor.

[0158] Specifically, in S505, the amplitude of the pixel point can be determined by the following formula (6), and the direction of the pixel point can be determined by the following formula (7):

[0159]

[0160] Among them, M is the amplitude of the pixel, and θ is the direction of the pixel.

[0161] Reference Figure 6 In some other embodiments, in order to generate feature descriptors more conveniently, quickly and accurately, S506 may specifically include S5061-S5063:

[0162] S5061, dividing the local pixel area into blocks to obtain a plurality of pixel blocks;

[0163] S5062, for each pixel block, counting direction histograms corresponding to a plurality of pixel points in the pixel block;

[0164] S5063, counting the distribution histogram corresponding to each pixel block to obtain a feature descriptor for each local pixel area.

[0165] In this embodiment, in the process of generating a feature descriptor, the local pixel area can be first divided into blocks in an N*N manner to obtain N*N pixel blocks, and then the direction histogram of the feature descriptor is determined for the amplitude and direction of each pixel block. The direction histogram reflects the gradient direction distribution information about the pixel extreme points in the pixel block.

[0166] In S5062, the direction histogram may be determined by the following formula (8):

[0167]

[0168] In this embodiment, H i.j is the direction histogram, M(x, y) is the amplitude information corresponding to the pixel extreme point, and histogram_bin(θ(x, y)) is the direction information corresponding to the pixel extreme point.

[0169] In this embodiment, the direction histograms of all pixel blocks are spliced ​​and merged into a feature descriptor, and normalized to obtain a local feature vector of the local pixel area.

[0170] As an example, the feature descriptor can be expressed by the following formulas (9) and (10):

[0171] D k =[H 1，1 , H 1，2 , ..., H N，N ] (9)

[0172]

[0173] Among them, formula (10) is the normalized feature descriptor.

[0174] In some other embodiments, by splicing all feature descriptors in a preset row direction, key local features of the video frame image at each scale are obtained to form a matrix Q:

[0175]

[0176] Reference Figure 7 In some other embodiments, the clustering result includes at least one target cluster. In order to obtain a more accurate target cluster, S103 may include S1031-S1035:

[0177] S1031, determining a plurality of second local ranges based on a plurality of key local features and a preset local radius;

[0178] S1032, determining a first local density and a first average distance of the target local feature within the second local range;

[0179] S1033. Determine the clustering score of the target local feature based on the first local density and the first average distance.

[0180] S1034. Determine at least one initial clustering kernel based on the clustering score.

[0181] S1035. Cluster the key local features within the second local range based on the initial clustering kernel and a preset distance threshold to obtain at least one target clustering cluster.

[0182] Since using the traditional k-means clustering method will have great unpredictability, in this embodiment, when determining the initial clustering kernel, in order to reduce unpredictability, first determine the second local range according to multiple key local features and a preset local radius. As an example, the second local range includes a key local feature and at least one second recognition feature, and the second recognition feature includes the recognition features other than the key local feature among the multiple first recognition features.

[0183] Then, determine the clustering score of the target local feature by determining the first local density and the first average distance between each target local feature and the second local feature, determine at least one initial clustering kernel according to the clustering score, and then determine the target clustering cluster through the clustering kernel and the preset distance threshold, where the target local feature includes the key local feature or the second local feature.

[0184] In this embodiment, the target clustering cluster determined through the above steps can reduce unpredictability, so that the obtained target clustering cluster is more accurate.

[0185] Specifically, in S1032, the first local density can be determined by the following formula (11):

[0186]

[0187] where ρ is the first local density, Ⅱ is the indicator function, which is 1 when d(x i , x j ) < r, and 0 otherwise.

[0188] The first average distance can be determined by the following formula (12):

[0189]

[0190] where δ i is the first average distance, and N is the data of the second recognition feature in the second local range.

[0191] In this embodiment, in S1033, the clustering score can be determined by the following formula (13):

[0192] S i =ρ i *δ i (13)

[0193] Among them, S i is the clustering score, ρ i is the first local density, δ i is the first average distance.

[0194] In this embodiment, in S1034, a preset number of target local features with the highest clustering scores may be selected to be determined as initial clustering cores.

[0195] In some embodiments, S1035 may specifically include:

[0196] Determine the first distance between each key local feature and the initial clustering core respectively, the initial clustering core is the initial clustering core with the shortest distance to the key local feature;

[0197] Based on the first distance and a preset distance threshold, an initial clustering cluster is determined, and the initial clustering cluster is determined as a target clustering cluster.

[0198] In this embodiment, multiple first recognition features may be clustered by calculating the Euclidean distance between each first recognition feature and an initial clustering kernel to obtain a target cluster.

[0199] Specifically, the Euclidean distance can be determined by the following formula (14):

[0200]

[0201] Among them, x j is the distance value of the first recognition feature in the jth dimension, μ ij is the initial clustering kernel μ i The distance value in the j-th dimension.

[0202] In some other embodiments, in order to ensure that the obtained target cluster is more accurate, after S1034, the method may further include:

[0203] Count the number of key local features of the initial clustering clusters;

[0204] Determine a first distance mean of each initial clustering cluster based on the key local feature vector and the first distance;

[0205] Based on the first distance mean, determining an updated clustering kernel of the initial clustering cluster;

[0206] In the case that the updated clustering core does not meet the preset stop updating condition, the step of respectively determining the first distance between each key local feature and the initial clustering core, wherein the initial clustering core is the initial clustering core with the closest distance to the key local feature, is repeated until the updated clustering core meets the preset stop updating condition, and the updated clustering core is determined as the target clustering core;

[0207] Based on the target clustering kernel, the initial clustering cluster is updated to obtain the target clustering cluster.

[0208] In this embodiment, the clustering effect for each initial clustering cluster may not be obvious, so the initial clustering cluster needs to be continuously updated, that is, the mean of each key local feature is continuously calculated to obtain an updated clustering core, so as to continuously update the clustering core to achieve continuous optimization and precise clustering effect.

[0209] Specifically, the number of key local features in each initial clustering cluster can be continuously counted, and then the first distance mean between each key local feature and the clustering core can be continuously calculated, and the initial clustering core can be continuously updated by the average value of the first distance mean to obtain an updated clustering core. When the updated clustering core and each first distance mean no longer change or a preset number of iterations is reached, the current updated clustering core is determined as the target clustering core, and then the initial clustering cluster is reconstructed based on the target clustering core to obtain the target clustering cluster.

[0210] In some other embodiments, since the feedback label only represents a part of the image recognition results when the user reviews the image recognition results, it is also necessary to label the remaining image recognition results based on the above target clustering clusters to obtain the final model fine-tuning samples. Therefore, S103 may also include:

[0211] Acquire multiple second identification features in the second identification result;

[0212] For each second identification feature:

[0213] Calculating a first similarity between the second identification feature and the first cluster;

[0214] Calculating a second similarity between the second identification feature and the second cluster;

[0215] When the first similarity is smaller than the first preset similarity and the second similarity is larger than the second preset similarity, the second identification feature corresponding to the first similarity is classified into the second cluster.

[0216] In this embodiment, the image recognition result includes a first recognition result and a second recognition result, and the feedback label corresponds to the first recognition result, that is, the first recognition result is the recognition result manually reviewed by the user, and the second recognition result is the recognition result not reviewed by the user; the feedback label includes at least one incorrectly recognized target feature and / or at least one correctly recognized target feature; the clustering result includes a first clustering cluster corresponding to the incorrectly recognized target feature and a second clustering cluster corresponding to the correctly recognized target feature.

[0217] In order to further improve the image recognition results, it is also necessary to cluster the second recognition results to obtain a complete model fine-tuning sample.

[0218] Specifically, the second identification feature can be calculated for similarity with the first cluster and the second cluster respectively to obtain a first similarity and a second similarity, and then the first similarity is compared with the first preset similarity, the second similarity and the second preset similarity to determine which cluster the second identification feature belongs to.

[0219] Finally, after all the second recognition features are classified into corresponding clusters, the recognition results of the second clusters are retained, and model fine-tuning samples are generated through the second clusters and feedback labels to make the trained image recognition model more accurate.

[0220] In some embodiments, the present application also provides a model fine-tuning device 800, which may include:

[0221] An acquisition module 801 is used to acquire an image recognition result of an image recognition model identifying video data and a user's feedback label on the image recognition result, wherein the image recognition result includes a plurality of first recognition features;

[0222] A feature extraction module 802 is used to extract key features of the video data based on the feedback tags to obtain multiple key local features;

[0223] A feature clustering module 803 is used to perform feature clustering on a plurality of first recognition features in the image recognition result based on a plurality of key local features, and obtain a clustering result corresponding to each key local feature, wherein the clustering result is used to characterize a recognition feature cluster having the same feedback label as the key local feature;

[0224] A construction module 804 is used to construct a model fine-tuning sample based on the clustering results and the corresponding feedback labels;

[0225] The fine-tuning training module 805 is used to perform fine-tuning training on the image recognition model according to the model fine-tuning samples to obtain a trained image recognition model.

[0226] As an optional implementation, the feature extraction module 802 may include:

[0227] An extraction unit, used for extracting video frame images from the video data to obtain a plurality of video frame images;

[0228] A construction unit, used to construct a scale space of each video frame image, the scale space includes video frame images at multiple scales, and the scale is used to represent different clarity levels of the video frame image;

[0229] A determination unit is used to determine a plurality of pixel extreme value points of the video frame image at each scale, wherein the pixel extreme value point represents a pixel point corresponding to a maximum pixel response value or a pixel point corresponding to a minimum pixel response value at the scale, and the pixel response value reflects the stability and importance of the pixel point;

[0230] Determine the feature descriptor corresponding to each pixel extreme point respectively, the feature descriptor is used to represent the local feature vector of the video frame image;

[0231] The splicing unit is used to splice the feature descriptors based on a preset row direction to obtain multiple key local features of the video frame image at each scale.

[0232] As an optional implementation manner, the determining unit is further specifically configured to:

[0233] Determine the local curvature of each pixel point based on a preset curvature calculation formula;

[0234] For each pixel point, based on the local curvature corresponding to the pixel point and a preset constant, determine a pixel response value of the pixel point;

[0235] Determine pixel extreme points from pixel points corresponding to multiple pixel response values.

[0236] As an optional implementation manner, the determining unit is further specifically configured to:

[0237] Based on the local curvature and pixel response value, each pixel is processed with sub-pixel precision to obtain a sub-pixel offset vector;

[0238] Determine key pixel points based on the sub-pixel offset vector;

[0239] Determine pixel extreme points from key pixel points corresponding to multiple pixel response values.

[0240] As an optional implementation manner, the determining unit is further specifically configured to:

[0241] An image coordinate system is established based on the video frame images at each scale.

[0242] For each pixel extreme point, obtain the pixel coordinates corresponding to the pixel extreme point from the image coordinate system;

[0243] Determine a local pixel area based on the pixel coordinates and the first preset local range, where the local pixel area includes a plurality of pixel points;

[0244] In the local pixel area, based on a preset gradient algorithm, determining a first gradient value in the horizontal direction and a second gradient value in the vertical direction corresponding to each pixel point in the local pixel area;

[0245] Determine the magnitude and direction corresponding to the pixel extreme point based on the first gradient value and the second gradient value;

[0246] Based on the amplitude and direction corresponding to the pixel extreme points, a feature descriptor of the local pixel area is generated.

[0247] As an optional implementation manner, the determining unit is further specifically configured to:

[0248] Dividing the local pixel area into blocks to obtain a plurality of pixel blocks;

[0249] For each pixel block, a direction histogram corresponding to multiple pixel points in the pixel block is counted, and the direction histogram reflects the gradient direction distribution information about the pixel extreme point in the pixel block;

[0250] The distribution histogram corresponding to each pixel block is counted to obtain the feature descriptor of each local pixel area.

[0251] As an optional implementation, the clustering result includes at least one target cluster, and the feature clustering module 803 can also be used for:

[0252] Based on the multiple key local features and the preset local radius, determine multiple second local ranges, the second local ranges include a key local feature and at least one second identification feature, the second identification feature includes the identification features of the multiple first identification features except the key local feature;

[0253] Determine, within the second local range, a first local density and a first average distance of a target local feature, where the target local feature includes a key local feature or a second local feature;

[0254] Determine a clustering score of the target local feature based on the first local density and the first average distance;

[0255] Based on the clustering score, determining at least one initial clustering core;

[0256] Based on the initial clustering kernel and the preset distance threshold, the key local features within the second local range are clustered to obtain at least one target cluster.

[0257] As an optional implementation, the feature clustering module 803 may also be used for:

[0258] Determine the first distance between each key local feature and the initial clustering core respectively, the initial clustering core is the initial clustering core with the shortest distance to the key local feature;

[0259] Based on the first distance and a preset distance threshold, an initial clustering cluster is determined, and the initial clustering cluster is determined as a target clustering cluster.

[0260] As an optional implementation, the feature clustering module 803 may also be used to: count the number of key local features of the initial clusters;

[0261] Determine a first distance mean of each initial clustering cluster based on the key local feature vector and the first distance;

[0262] Based on the first distance mean, determining an updated clustering kernel of the initial clustering cluster;

[0263] In the case that the updated clustering core does not meet the preset stop updating condition, the step of respectively determining the first distance between each key local feature and the initial clustering core, wherein the initial clustering core is the initial clustering core with the closest distance to the key local feature, is repeated until the updated clustering core meets the preset stop updating condition, and the updated clustering core is determined as the target clustering core;

[0264] Based on the target clustering kernel, the initial clustering cluster is updated to obtain the target clustering cluster.

[0265] As an optional implementation, the image recognition result includes a first recognition result and a second recognition result, and the feedback tag corresponds to the first recognition result;

[0266] The feedback tag includes at least one incorrectly identified target feature and / or at least one correctly identified target feature; the clustering result includes a first cluster corresponding to the incorrectly identified target feature and a second cluster corresponding to the correctly identified target feature; the feature clustering module 803 can also be used to:

[0267] Acquire multiple second identification features in the second identification result;

[0268] For each second identification feature:

[0269] Calculating a first similarity between the second identification feature and the first cluster;

[0270] Calculating a second similarity between the second identification feature and the second cluster;

[0271] When the first similarity is smaller than the first preset similarity and the second similarity is larger than the second preset similarity, the second identification feature corresponding to the first similarity is classified into the second cluster.

[0272] Fig. 9A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0273] The electronic device may include a processor 901 and a memory 902 storing computer program instructions.

[0274] Specifically, the processor 901 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0275] The memory 902 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 902 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 902 may include a removable or non-removable (or fixed) medium, or the memory 902 is a non-volatile solid-state memory. The memory 902 may be inside or outside the integrated gateway disaster recovery device.

[0276] In one example, the memory 902 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0277] The memory 902 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the model fine-tuning method according to the first aspect of the present disclosure.

[0278] The processor 901 reads and executes the computer program instructions stored in the memory 902 to implement Fig. 9 A video query method in the illustrated embodiment.

[0279] In one example, the electronic device may further include a communication interface 903 and a bus 904. Fig. 9 As shown, the processor 901, the memory 902, and the communication interface 903 are connected via a bus 904 and communicate with each other.

[0280] The communication interface 903 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0281] Bus 904 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), hypertransmission (Hyper Transport, HT) interconnection, industry standard architecture (IndustryStandard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 904 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0282] The electronic device can execute the model fine-tuning method in the embodiment of the present application, thereby realizing the combination Figure 1-Figure 8 Describe the model fine-tuning method and apparatus.

[0283] In addition, in combination with the model fine-tuning method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the model fine-tuning methods in the above embodiment is implemented.

[0284] In an optional embodiment, in combination with the model fine-tuning method in the above embodiment, the embodiment of the present application can provide a computer program product for implementation, and the instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device can implement any one of the model fine-tuning methods in the above embodiment.

[0285] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0286] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0287] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0288] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0289] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A model fine-tuning method, characterized in that: include: Obtaining an image recognition result of an image recognition model identifying video data and a user's feedback label on the image recognition result, wherein the image recognition result includes a plurality of first recognition features; Extract key features from the video data based on the feedback tags to obtain multiple key local features; Performing feature clustering on the multiple first recognition features in the image recognition result based on the multiple key local features to obtain a clustering result corresponding to each of the key local features, wherein the clustering result is used to characterize a recognition feature cluster having the same feedback label as the key local feature; Based on the clustering results and the corresponding feedback labels, construct a model fine-tuning sample; The image recognition model is fine-tuned and trained according to the model fine-tuning samples to obtain a fine-tuned image recognition model.

2. The method according to claim 1, characterized in that The extracting key features of the video data based on the feedback tag to obtain a plurality of key local features includes: Extracting video frame images from the video data to obtain multiple video frame images; Constructing a scale space of each of the video frame images, wherein the scale space includes video frame images at multiple scales, and the scales are used to represent different clarity levels of the video frame images; For each video frame image at the scale, determine a plurality of pixel extreme value points of the video frame image, wherein the pixel extreme value points represent pixel points corresponding to the maximum pixel response value or the minimum pixel response value at the scale, and the pixel response value reflects the stability and importance of the pixel points; Determine the feature descriptor corresponding to each of the pixel extreme points respectively, wherein the feature descriptor is used to represent the local feature vector of the video frame image; The feature descriptors are spliced ​​based on a preset row direction to obtain a plurality of key local features of the video frame image at each scale.

3. The method according to claim 2, characterized in that The step of determining a plurality of pixel extreme value points of the video frame image comprises: Determine the local curvature of each pixel point based on a preset curvature calculation formula; For each of the pixel points, determining a pixel response value of the pixel point based on a local curvature corresponding to the pixel point and a preset constant; A pixel extreme point is determined from the pixel points corresponding to the plurality of pixel response values.

4. The method according to claim 3, characterized in that Before determining the pixel extreme value point from the pixel points corresponding to the plurality of pixel response values, the method further comprises: Based on the local curvature and the pixel response value, performing sub-pixel precision processing on each pixel point to obtain a sub-pixel offset vector; Determining a key pixel point based on the sub-pixel offset vector; Determining a pixel extreme point from the pixel points corresponding to the plurality of pixel response values ​​comprises: A pixel extreme point is determined from the key pixel points corresponding to the plurality of pixel response values.

5. The method according to claim 2, characterized in that: The step of respectively determining the feature descriptor corresponding to each of the pixel extreme points comprises: Establish an image coordinate system based on the video frame image at each scale; For each of the pixel extreme value points, obtaining the pixel coordinates corresponding to the pixel extreme value point from the image coordinate system; Based on the pixel coordinates and a first preset local range, determining a local pixel area, wherein the local pixel area includes a plurality of pixel points; In the local pixel area, based on a preset gradient algorithm, determining a first gradient value in the horizontal direction and a second gradient value in the vertical direction corresponding to each pixel point in the local pixel area; Determine the amplitude and direction corresponding to the pixel extreme point based on the first gradient value and the second gradient value; A feature descriptor of the local pixel area is generated based on the amplitude and direction corresponding to the pixel extreme point.

6. The method according to claim 5, characterized in that The determining of the feature descriptor of each local pixel region based on the amplitude and direction corresponding to the pixel point includes: Dividing the local pixel area into blocks to obtain a plurality of pixel blocks; For each of the pixel blocks, counting direction histograms corresponding to a plurality of the pixel points in the pixel block, the direction histogram reflecting gradient direction distribution information about the pixel extreme value points in the pixel block; The distribution histogram corresponding to each pixel block is counted to obtain a feature descriptor of each local pixel area.

7. The method according to any one of claims 1 to 6, characterized in that: The clustering result includes at least one target cluster, and the clustering of the multiple first recognition features in the image recognition result based on the multiple key local features to obtain the clustering result corresponding to each of the key local features includes: Based on the plurality of key local features and the preset local radius, a plurality of second local ranges are determined, wherein the second local ranges include a key local feature and at least one second identification feature, and the second identification feature includes identification features of the plurality of first identification features except the key local feature; Determine, within the second local range, a first local density and a first average distance of a target local feature, wherein the target local feature includes a key local feature or a second local feature; Determining a clustering score of the target local feature based on the first local density and the first average distance; Based on the clustering score, determining at least one initial clustering core; Based on the initial clustering kernel and a preset distance threshold, the key local features within the second local range are clustered to obtain at least one target cluster.

8. The method according to claim 7, characterized in that The step of clustering the key local features within the second local range based on the initial clustering kernel and a preset distance threshold to obtain at least one target clustering cluster includes: Determine a first distance between each of the key local features and the initial clustering core respectively, the initial clustering core being the initial clustering core that is closest to the key local features; Based on the first distance and the preset distance threshold, an initial cluster is determined, and the initial cluster is determined as the target cluster.

9. The method according to claim 8, characterized in that After determining the initial cluster based on the first distance and the preset distance threshold, the method further includes: Counting the number of key local features of the initial clustering clusters; Determine a first distance mean of each of the initial clusters based on the key local features and the first distance; Based on the first distance mean, determining an updated clustering core of the initial clustering cluster; In the case that the updated clustering core does not meet the preset stop updating condition, repeating the step of respectively determining the first distance between each of the key local features and the initial clustering core, wherein the initial clustering core is the initial clustering core with the closest distance to the key local features, until the updated clustering core meets the preset stop updating condition, and determining the updated clustering core as the target clustering core; Based on the target clustering core, the initial clustering cluster is updated to obtain a target clustering cluster.

10. The method according to any one of claims 1 to 9, characterized in that: The image recognition result includes a first recognition result and a second recognition result, and the feedback tag corresponds to the first recognition result; The feedback tag includes at least one incorrectly identified target feature and / or at least one correctly identified target feature; the clustering result includes a first cluster corresponding to the incorrectly identified target feature and a second cluster corresponding to the correctly identified target feature; The performing feature clustering on the plurality of first recognition features in the image recognition result based on the plurality of key local features to obtain a clustering result corresponding to each of the key local features further includes: Acquire a plurality of second identification features in the second identification result; For each second identification feature: Calculating a first similarity between the second identification feature and the first cluster; Calculating a second similarity between the second identification feature and the second cluster; When the first similarity is smaller than a first preset similarity and the second similarity is larger than a second preset similarity, the second identification feature corresponding to the first similarity is classified into the second cluster.

11. A model fine-tuning device, characterized in that: The device comprises: An acquisition module, used to acquire an image recognition result of an image recognition model identifying video data and a user's feedback label on the image recognition result, wherein the image recognition result includes a plurality of first recognition features; A feature extraction module, used to extract key features of the video data based on the feedback tags to obtain multiple key local features; A feature clustering module, used for performing feature clustering on the multiple first recognition features in the image recognition result based on the multiple key local features to obtain a clustering result corresponding to each of the key local features, wherein the clustering result is used to characterize a recognition feature cluster having the same feedback label as the key local feature; A construction module, used to construct a model fine-tuning sample based on the clustering results and the corresponding feedback labels; The fine-tuning training module is used to perform fine-tuning training on the image recognition model according to the model fine-tuning sample to obtain a trained image recognition model.

12. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the model fine-tuning method as described in any one of claims 1-10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the model fine-tuning method according to any one of claims 1 to 10 is implemented.

Images