Model training method, business processing method, device, terminal and storage medium

Abstract

The embodiment of the present invention discloses a model training method, a business processing method, an apparatus, a terminal and a medium, wherein the model training method comprises: obtaining a sample set, wherein the sample set comprises a plurality of sample images and image relationships, wherein the image relationships comprise an association relationship between at least two sample images; constructing a data structure based on the image relationships and the plurality of sample images; traversing each element of the data structure and clustering the sample images stored in each element to obtain a plurality of cluster sets and category information of the sample images in each cluster set; using the plurality of sample images and the category information of each sample image to perform model training on an image processing model; the embodiment of the present invention can improve the efficiency of model training and reduce labor costs.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, specifically to the field of image processing technology, and in particular to a model training method, a business processing method, a model training device, a business processing device, a terminal and a computer storage medium. Background Art

[0002] An image processing model is a model used to extract features from an input image. Currently, the model training method for the image processing model mainly includes the following process: first, a large number of sample images are obtained; second, professional labelers are hired to label each sample image to obtain the category label of each sample image; then, the sample images with category labels are used to train the image processing model. The inventors have found in practice that the above-mentioned model training method requires professional labelers to label the sample images, which has a high labor cost; and a lot of time is consumed in the entire model training process to label the category label of each sample image, which leads to low model training efficiency. Summary of the invention

[0003] The embodiments of the present invention provide a model training method, a business processing method, a device, a terminal and a computer storage medium, which can improve the efficiency of model training and reduce labor costs.

[0004] On the one hand, an embodiment of the present invention provides a model training method, the model training method comprising:

[0005] Acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images;

[0006] Constructing a data structure based on the image relationship and the multiple sample images; wherein the data structure is composed of multiple elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated relationship exists between the sample images stored in the associated element pair;

[0007] Traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same;

[0008] The image processing model is trained using the multiple sample images and the category information of each sample image.

[0009] On the other hand, an embodiment of the present invention provides a service processing method, the service processing method comprising:

[0010] In response to a service request, obtaining a target image to be processed;

[0011] Calling an image processing model to extract features of the target image to obtain a first feature of the target image; the image processing model is trained using the above-mentioned model training method;

[0012] Performing refinement processing according to the first feature of the target image to obtain a second feature of the target image;

[0013] The requested service is executed according to the second feature of the target image.

[0014] In another aspect, an embodiment of the present invention provides a model training device, the model training device comprising:

[0015] An acquisition unit, configured to acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images;

[0016] A construction unit, configured to construct a data structure based on the image relationship and the plurality of sample images; wherein the data structure is composed of a plurality of elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated relationship exists between the sample images stored in the associated element pair;

[0017] A clustering unit, used to traverse each element of the data structure and cluster the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same;

[0018] The training unit is used to use the multiple sample images and the category information of each sample image to perform model training on the image processing model.

[0019] In another aspect, an embodiment of the present invention provides a service processing device, the service processing device comprising:

[0020] An acquisition unit, used for acquiring a target image to be processed in response to a service request;

[0021] A processing unit, used for calling an image processing model to extract features of the target image to obtain a first feature of the target image; the image processing model is trained using the above-mentioned model training method;

[0022] The processing unit is used to perform refinement processing according to the first feature of the target image to obtain the second feature of the target image;

[0023] A service unit is used to execute the requested service according to the second feature of the target image.

[0024] In another aspect, an embodiment of the present invention provides a terminal, the terminal including an input device and an output device, the terminal further including:

[0025] a processor adapted to implement one or more instructions; and,

[0026] A computer storage medium storing one or more first instructions, wherein the one or more first instructions are suitable for being loaded by the processor and executing the following steps:

[0027] Acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images;

[0028] Constructing a data structure based on the image relationship and the multiple sample images; wherein the data structure is composed of multiple elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated relationship exists between the sample images stored in the associated element pair;

[0029] Traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same;

[0030] The image processing model is trained using the multiple sample images and the category information of each sample image.

[0031] Alternatively, the computer storage medium stores one or more second instructions, and the one or more second instructions are suitable for being loaded by the processor and executing the following steps:

[0032] In response to a service request, obtaining a target image to be processed;

[0033] Calling an image processing model to extract features of the target image to obtain a first feature of the target image; the image processing model is trained using the above-mentioned model training method;

[0034] Performing refinement processing according to the first feature of the target image to obtain a second feature of the target image;

[0035] The requested service is executed according to the second feature of the target image.

[0036] In another aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer storage medium stores one or more first instructions, wherein the one or more first instructions are suitable for being loaded by a processor and executing the following steps:

[0037] Acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images;

[0038] Constructing a data structure based on the image relationship and the multiple sample images; wherein the data structure is composed of multiple elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated relationship exists between the sample images stored in the associated element pair;

[0039] Traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same;

[0040] The image processing model is trained using the multiple sample images and the category information of each sample image.

[0041] Alternatively, the computer storage medium stores one or more second instructions, and the one or more second instructions are suitable for being loaded by the processor and executing the following steps:

[0042] In response to a service request, obtaining a target image to be processed;

[0043] Calling an image processing model to extract features of the target image to obtain a first feature of the target image; the image processing model is trained using the above-mentioned model training method;

[0044] Performing refinement processing according to the first feature of the target image to obtain a second feature of the target image;

[0045] The requested service is executed according to the second feature of the target image.

[0046] In the process of model training, the embodiment of the present invention can first obtain a sample set, and construct a data structure based on the image relationship and the multiple sample images. Since any two connected elements in the data structure are associated element pairs, there is an association relationship between the sample images stored in the associated element pairs, so each element of the data structure can be traversed and the sample images stored in each element can be clustered to obtain multiple cluster sets and the category information of the sample images in each cluster set; the category information of the sample images is obtained by using the data structure, and no manual labeling is required by the labeling personnel, which can improve the efficiency of model training and reduce labor costs. After obtaining the category information of each sample image, multiple sample images and the category information of each sample image can be used to train the image processing model; it can be seen that the embodiment of the present invention can also effectively convert the unsupervised model training task into a supervised model training task, so that the image processing model can better learn the image features in the sample image, thereby improving the accuracy of the image processing model; and by constructing a sample image training image processing model with category information, the feature expression ability of the image processing model for simple texture images can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0048] Figure 1 It is a flowchart of a model training solution provided by an embodiment of the present invention;

[0049] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present invention;

[0050] Figure 3 It is an image schematic diagram of a homologous trademark group provided by an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of a data structure provided by an embodiment of the present invention;

[0052] Figure 5 It is a schematic diagram of constructing an undirected graph provided by an embodiment of the present invention;

[0053] Figure 6 is a flowchart of an image processing method provided by another embodiment of the present invention;

[0054] Figure 7a is a node schematic diagram of a connectivity graph provided by an embodiment of the present invention;

[0055] Figure 7b is an image schematic diagram of a connectivity graph provided by an embodiment of the present invention;

[0056] Figure 8 It is a flowchart of a service processing method provided by an embodiment of the present invention;

[0057] Fig. 9 It is a flowchart of a refined processing provided by an embodiment of the present invention;

[0058] Fig.10a It is a flowchart of a business process provided by an embodiment of the present invention;

[0059] Fig.10b is an application scenario diagram of an image processing model provided by an embodiment of the present invention;

[0060] Fig.10c is another application scenario diagram of an image processing model provided by an embodiment of the present invention;

[0061] Fig.10d is a schematic diagram of a trademark search result provided by an embodiment of the present invention;

[0062] Fig.11 is a structural schematic diagram of a model training device provided by an embodiment of the present invention;

[0063] Fig.12 is a structural diagram of a service processing device provided by an embodiment of the present invention;

[0064] Fig.13 It is a schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0066] An image processing model is a model used to extract features from an input image. The embodiment of the present invention proposes a model training scheme for the image processing model. The accuracy of the image processing model and the model training efficiency can be improved by using the model training scheme to train the image processing model. The model training scheme can be applied to a terminal, which includes but is not limited to: a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. When the terminal executes the model training scheme, its training process can be seen in Figure 1As shown, it may specifically include the following two parts: ① Cluster multiple sample images to obtain category information of each sample image. Specifically, the terminal may first obtain multiple sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images. Secondly, a data structure may be constructed based on image relationships and multiple sample images; the data structure here refers to a set of data elements that have one or more specific relationships with each other, which can be used to store data; the data structure may include but is not limited to: an undirected graph, a directed graph, a tree diagram, etc. Then, the data structure may be traversed, and clustering may be performed to obtain multiple cluster sets and category information of sample images in each cluster set. ② Perform model training on the image processing model. Specifically, multiple sample images and category information of each sample image may be used, and the image processing model may be trained based on the loss function of the image processing model.

[0067] It can be seen that the model training scheme proposed in the embodiment of the present invention can use the data structure to obtain the category information of each sample image, without the need for manual labeling by labeling personnel, which can reduce labor costs; using each sample image and the category image of each sample image for model training can effectively convert the unsupervised model training task into a supervised model training task, thereby improving the accuracy of the image processing model.

[0068] Based on the above description, an embodiment of the present invention proposes a model training method, which can be executed by a terminal. Figure 2 As shown, the model training method may include the following steps S201-S204:

[0069] S201, obtaining a sample set.

[0070] The sample set may include multiple sample images and image relationships, and the image relationship includes an association relationship between at least two sample images, and the image relationship may be in the form of NN. The sample images may include but are not limited to: trademark images, face images, landscape images, etc.; the association relationship between two sample images refers to a relationship used to reflect the similarity between the two sample images; if there is an association relationship between two sample images, it indicates that the similarity between the two sample images is large (greater than the similarity threshold).

[0071] In the specific implementation process of obtaining the sample set, the terminal can obtain multiple sample images and the association relationship between at least two sample images; add the association relationship between at least two sample images to the image relationship, and use the image relationship and multiple sample images to construct a sample set. In one embodiment, the association relationship between the at least two sample images can be obtained based on manual annotation. In another embodiment, the association relationship between the at least two sample images can also be obtained based on a similar image list, which includes multiple similar image groups, and the similarity between any two images in each similar image group is greater than the similarity threshold. Specifically, for any two sample images (for example, the first sample image and the second sample image), if there is a similar image group in the similar image list that includes the first sample image and the second sample image, it indicates that there is an association relationship between the first sample image and the second sample image. Optionally, when the sample image is a trademark image, the similar image list can be a homologous trademark list, which includes multiple homologous trademark image groups; the similarity between any two trademark images in each homologous trademark image group is greater than the similarity threshold, and any trademark image in the same homologous trademark image group can be called the homologous trademark of the remaining trademark images. For example, suppose the same-source trademark image group A includes trademark image 1 and trademark image 2. Figure 3 As shown; then trademark image 1 can be called the homologous trademark of trademark image 2, and trademark image 2 can also be called the homologous trademark of trademark image 1.

[0072] S202: construct a data structure based on the image relationship and multiple sample images.

[0073] The data structure is composed of multiple elements, one element stores a sample image, any two connected elements are associated element pairs, and there is an association relationship between the sample images stored in the associated element pairs; the data structure may include but is not limited to: undirected graph, directed graph, tree graph, etc. Figure 4 As shown in the figure: an undirected graph G(V, E) refers to a graph composed of multiple nodes (i.e. elements) and undirected edges, where V is a sample set and E is an image relationship; a directed graph refers to a graph composed of multiple nodes (i.e. elements) and directed edges; a tree diagram refers to a graphical representation of a data tree, which is a graph based on a parent-child hierarchy and composed of multiple nodes (i.e. elements).

[0074] In the specific implementation process of step S202, the terminal may first use multiple elements to store multiple sample images, with one element storing one sample image; secondly, determine the associated element pairs from the multiple elements based on the image relationship; and then connect the associated element pairs to obtain a data structure. Taking the data structure as an undirected graph as an example, see Figure 5As shown in the figure: the sample set includes 5 sample images, namely sample image 1, sample image 2, sample image 3, sample image 4 and sample image 5; the image relationship includes: the association relationship between sample image 1 and sample image 3, the association relationship between sample image 2 and sample image 3, the association relationship between sample image 2 and sample image 4, and the association relationship between sample image 4 and sample image 5. First, 5 nodes (i.e., elements) can be used to store the 5 sample images, and one node stores one sample image; specifically, it can be expressed as: node 1 stores sample image 1, node 2 stores sample image 2, node 3 stores sample image 3, node 4 stores sample image 4, and node 5 stores sample image 5. Secondly, according to the image relationship, it can be determined that the associated node pairs include: the node pair composed of node 1 and node 3, the node pair composed of node 2 and node 3, the node pair composed of node 2 and node 4, and the node pair composed of node 4 and node 5. Then, the associated node pairs can be connected to obtain an undirected graph.

[0075] S203, traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set.

[0076] From the above, it can be seen that any two connected elements in the data structure are associated element pairs, and there is an association relationship between the sample images stored in the associated element pairs, that is, the similarity between the sample images stored in any two connected elements in the data structure is relatively large. Practice has shown that two images with a large similarity (that is, two images with an association relationship) are usually images of the same category; therefore, the terminal can regard the sample images stored in any two connected elements in the data structure as images of the same category. Moreover, practice has further shown that if two sample images are respectively associated with the same sample image, then the two sample images are usually also images of the same category; for example, image a and image b are associated, and image b and image c are associated, then image a, image b and image c can be considered to be images of the same category. Therefore, the terminal can also regard the sample images stored in each element connected to the same element in the data structure as images of the same category; for example Figure 5 As shown in the figure, node 1 and node 2 are connected to node 3 respectively, so it can be considered that sample image 1 stored in node 1 and sample image 2 stored in node 2 are images of the same category. Based on this, the terminal can traverse each element of the data structure and cluster the sample images stored in each element to obtain multiple clustering sets; the so-called traversal means that each element in the data structure is accessed once and only once in sequence along a certain search route.

[0077] In one implementation, the terminal may traverse each element of the data structure and cluster the sample images stored in the elements on the same connected path into the same cluster set; wherein the connected path refers to a path in which any element in the path can be connected to another element. Figure 4 Take the undirected graph in as an example, in path 1 formed by element 1-element 4, any element can be connected to another element, so path 1 is a connected path; and in path 2 formed by element 1-element 5, element 1 and element 5 are not connected, so path 2 is a non-connected path. In another embodiment, practice shows that the greater the element depth between two elements, the smaller the similarity of the sample images stored by the two elements, and at this time the sample images stored by the two elements may not be images of the same category. Therefore, the terminal can also traverse each element of the data structure based on the depth restriction condition, so as to cluster the sample images stored by each element to obtain multiple clustering sets; the depth restriction condition here includes: if the element depth of the current element being accessed in the current traversal process is greater than a preset threshold, the current traversal process is terminated. Among them, the preset threshold can be set according to an empirical value or actual needs; the element depth of the current element is determined according to the number of edges between the current element and the starting element in the current traversal process; for example Figure 5 As shown, the starting element (node ​​2) and the current element (node ​​5) in the current traversal process are connected through node 4, then the number of edges between the current element and the starting element is 2, and the element depth of the current element is 2. Since traversing the data structure requires accessing all elements in the data structure, after the current traversal process ends, the next traversal process can be started to access the remaining elements that have not been accessed in the data structure. It can be seen that one or more traversal processes may be started when traversing each element of the data structure based on the depth restriction condition, and one or more elements can be accessed in each traversal process; then the sample images stored in the elements accessed in the same traversal process can be clustered into the same cluster set, thereby obtaining multiple cluster sets. After obtaining multiple cluster sets, the same category information can be annotated for the sample images in the same cluster set, thereby obtaining the category information of the sample images in each cluster set; the category information of the sample images in each cluster set is the same. Optionally, when traversing each element of the data structure based on the depth restriction condition, the terminal can also traverse each element of the data structure based on the depth restriction condition and use the BFS algorithm (Breadth First Search, breadth first algorithm).

[0078] S204, using multiple sample images and category information of each sample image to perform model training on the image processing model.

[0079] The image processing model may include a convolutional layer and a fully connected layer. The image processing model may be a model constructed based on CNN (Convolutional Neural Networks), such as: VGGNet network (Visual Geometry Group Network, a convolutional neural network), ResNet network (residual network) and AlexNet network (a convolutional neural network), etc.; it may also be a model constructed based on FCN (Fully Convolutional Networks, full neural network), which is not limited. In the specific implementation process of step S204, the terminal may first call the image processing model to obtain the image features of each sample image, and perform category prediction based on the image features of each sample image to obtain the prediction information of each sample image; then, according to the difference between the prediction information of each sample image and the category information of each sample image, adjust the parameters of the image processing model, so as to realize model training of the image processing model.

[0080] In the process of model training, the embodiment of the present invention can first obtain a sample set, and construct a data structure based on image relationships and multiple sample images. Since any two connected elements in the data structure are associated element pairs, there is an association relationship between the sample images stored in the associated element pairs, so each element of the data structure can be traversed and the sample images stored in each element can be clustered to obtain multiple cluster sets and category information of sample images in each cluster set; the category information of the sample images is obtained by using the data structure, and no manual labeling is required by the labeling personnel, which can improve the efficiency of model training and reduce labor costs. After obtaining the category information of each sample image, multiple sample images and the category information of each sample image can be used to train the image processing model; it can be seen that the embodiment of the present invention can also effectively convert the unsupervised model training task into a supervised model training task, so that the image processing model can better learn the image features in the sample image, thereby improving the accuracy of the image processing model; and by constructing a sample image training image processing model with category information, the feature expression ability of the image processing model for simple texture images can also be improved.

[0081] See also Figure 6 , is a flow chart of another model training method provided by an embodiment of the present invention, and the model training method can be executed by a terminal. The model training method may include the following steps S601-S606:

[0082] S601, obtaining a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images.

[0083] S602: construct a data structure based on the image relationship and multiple sample images.

[0084] S603, traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same.

[0085] As can be seen from the foregoing, the data structure may include an undirected graph, a directed graph, a tree graph, etc.; taking the case where the data structure includes an undirected graph and the elements of the data structure are nodes of the undirected graph as an example, the specific implementation of step S603 may include the following steps s11-s12:

[0086] s11, traverse each node of the undirected graph based on the depth restriction condition to obtain multiple connected graphs.

[0087] In the specific implementation process, step s11 may include: ① Select any unvisited node from the undirected graph as the starting node of the current traversal process, and visit the starting node. ② Based on the node width-first traversal principle and along the traversal direction from shallow to deep node depth, visit the remaining nodes in the undirected graph except the starting node and not visited; the node width-first traversal principle here refers to the principle of giving priority to traversing nodes of the same node depth, and the traversal direction from shallow to deep node depth refers to the traversal direction from small to large node depth, that is, first traverse the nodes with a node depth of 1, and then traverse the nodes with a node depth of 2... and so on. For example Figure 5 In the undirected graph shown, nodes 1 to 5 have not been visited. If the starting node of the current traversal process is node 3, nodes 2 and 1 of the same node depth (both node depths are 1) are traversed first. Specifically, node 2 can be traversed first and then node 1, or node 1 can be traversed first and then node 2. After traversing nodes 1 and 2, the remaining nodes are traversed. Since the node depth of node 4 among the remaining nodes is 2 and the node depth of node 5 is 3, that is, the node depth of node 4 is shallower than that of node 5, when traversing the remaining nodes, node 4 is traversed first and then node 5.

[0088] It should be noted that when traversing each node, if the current node being traversed has been visited, the current node being traversed will no longer be visited, and the traversal will continue to the next node. In addition, when visiting each node, it is necessary to determine whether the node depth of the current node being visited is greater than the preset threshold; if it is, execute step ③: If the node depth of the current node being visited is greater than the preset threshold, the current traversal process is terminated, and a connected graph is constructed using the nodes visited in the current traversal process; wherein, the node depth of the current node is determined according to the number of edges between the current node and the starting node. ④ Repeat the above steps (i.e., repeat the above steps ①-③) until all nodes of the undirected graph are visited, and multiple connected graphs are obtained; wherein, the node schematic diagrams of the multiple connected graphs can be found in Figure 7a As shown in Figure 2, each connectivity graph can correspond to an image diagram; taking the connectivity graph of cluster 3 as an example, its image diagram can be found in Figure 7b shown.

[0089] s12, adding the sample images stored in the nodes of the same connected graph to the same cluster set, and marking the same category information for the sample images in the same cluster set.

[0090] In the specific implementation process, after obtaining multiple connected graphs, the sample images stored in the nodes of the same connected graph can be added to the same cluster set, and the same category information can be annotated for the sample images in the same cluster set; that is, a cluster set can include one or more sample images, and these one or more sample images can be considered as images of the same category. It should be noted that the category information annotated for the sample image is not used to indicate the actual category of the sample image, but is only an identifier for distinguishing sample images in different cluster sets; for example, the actual category of sample image a is a landscape image, and the category information annotated for sample image a can be category 1; if the category information of sample image b is also category 1, it indicates that sample image a and sample image b belong to the same cluster set; if the category information of sample image b is category 2, it indicates that sample image a and sample image b belong to different cluster sets. It can be seen that when the embodiment of the present invention annotates the same category information for sample images in the same cluster set, it does not pay attention to the actual category of the sample image, and can randomly select a category information to annotate each sample image in the same cluster set, which can reduce the difficulty of annotating the category information of the sample image and improve the annotation efficiency.

[0091] S604, calling the image processing model to extract features from each sample image to obtain image features of each sample image.

[0092] S605, calling the image processing model to perform category prediction according to the image features of each sample image, and obtaining prediction information of each sample image.

[0093] In steps S604-S605, since the image processing model is a model for extracting features from images, the image processing model can be called to extract features from each sample image to obtain image features of each sample image. After obtaining the image features of each sample image, the image processing model can be called to perform category prediction based on the image features of each sample image to obtain prediction information of each sample image, which includes the probability value of the sample image on each category information.

[0094] S606: Perform model training on the image processing model according to the prediction information of each sample image and the category information of each sample image.

[0095] In the specific implementation process, the loss function of the image processing model can be obtained first, and the loss function includes: a feature distance loss function and a feature classification loss function; wherein, the feature distance loss function refers to a loss function used to promote the image processing model to learn the intra-class feature expression distance to become smaller and / or the inter-class feature expression distance to become larger, such as the CL (Center Loss, center loss) function, the maximum interval loss function, etc.; the feature classification loss function refers to a loss function that allows the image processing model to learn the category information of the image so that the feature expression of the image processing model has discriminability, such as the Softmax Loss function (normalized exponential function). The embodiment of the present invention is explained by taking the feature distance loss function as the CL function and the feature classification loss function as the Softmax Loss function as an example; wherein the CL function can be shown as Formula 1.1, and the Softmax Loss function can be shown as Formula 1.2:

[0096]

[0097]

[0098] In the above formulas 1.1 and 1.2, m is the number of sample images in the sample set used to train the image processing model; if the sample set includes a large number of sample images, the sample set can also be used for model training based on the mini-batch (batch training) method, that is, the sample set can be divided into multiple sample subsets, and each sample subset is used for model training in turn; then m in this case is the size of the mini-batch, that is, the number of sample images in each sample subset; x i is the feature vector of the last fully connected layer or convolutional layer of the i-th sample image in the image processing model, that is, the image feature of the i-th sample image; For the yth i The central feature vector of the y-th category information. iThe image feature vector of the sample image at the center position in each sample image corresponding to the category information; S ij is the probability value of the i-th sample image belonging to the j-th category information, y ij is the true label value of the i-th sample image belonging to the j-th category information; if the category information of the i-th sample image is the same as the j-th category information, then y ij The value of y is 1, otherwise ij The value of is 0.

[0099] After obtaining the loss function of the image processing model, the value of the loss function can be calculated according to the prediction information of each sample image and the category information of each sample image. Specifically, the prediction information of each sample image and the category information of each sample image can be substituted into the above formula 1.1 and formula 1.2 for calculation to obtain the loss value of the feature distance loss function and the loss value of the feature classification loss function; then the sum of the loss value of the feature distance loss function and the loss value of the feature classification loss function is obtained as the value of the loss function. After obtaining the value of the loss function, the image processing model can be optimized according to the principle of reducing the value of the loss function. The embodiment of the present invention jointly determines the value of the loss function according to the feature distance loss function and the feature classification loss function, so that when the image processing model is subsequently optimized according to the principle of reducing the value of the loss function, the parameters of the image processing model can be adjusted more accurately, thereby improving the accuracy of the image processing model.

[0100] In the process of model training, the embodiment of the present invention can first obtain a sample set, and construct a data structure based on image relationships and multiple sample images. Since any two connected elements in the data structure are associated element pairs, there is an association relationship between the sample images stored in the associated element pairs, so each element of the data structure can be traversed and the sample images stored in each element can be clustered to obtain multiple cluster sets and category information of sample images in each cluster set; the category information of the sample images is obtained by using the data structure, and no manual labeling is required by the labeling personnel, which can improve the efficiency of model training and reduce labor costs. After obtaining the category information of each sample image, multiple sample images and the category information of each sample image can be used to train the image processing model; it can be seen that the embodiment of the present invention can also effectively convert the unsupervised model training task into a supervised model training task, so that the image processing model can better learn the image features in the sample image, thereby improving the accuracy of the image processing model; and by constructing a sample image training image processing model with category information, the feature expression ability of the image processing model for simple texture images can also be improved.

[0101] After the model training of the image processing model is completed by using the above-mentioned model training method, the trained image processing model can also be used in different application scenarios according to business needs; for example: trademark search scenario, similar image search scenario, and image similarity calculation scenario, etc. Based on this, the embodiment of the present invention also provides a business processing method, which can be executed by the terminal. Figure 8 As shown, the service processing method may include the following steps S801-S804:

[0102] S801, in response to a service request, obtaining a target image to be processed.

[0103] The service request may include, but is not limited to, an image retrieval request, an image similarity calculation request, etc., wherein the image retrieval request may include a trademark retrieval request, a similar image retrieval request, etc. The service request may carry an image identifier of a target image; after detecting the service request, the terminal may respond to the service request and obtain a target image to be processed according to the image identifier carried in the service request; the target image here may be any image, such as a trademark image, a face image, or a landscape image, etc.

[0104] S802, calling the image processing model to extract features of the target image to obtain a first feature of the target image.

[0105] Among them, the image processing model is adopted Figure 2 or Figure 6 The specific training process can be found in the relevant description of the above-mentioned model training method embodiment, and will not be repeated here. After acquiring the target image to be processed, the terminal can call the image processing model to extract features of the target image to obtain the first feature of the target image; the feature dimension of the first feature can be expressed as c×h×w (c represents the number of channels of the first feature, h represents the height of the first feature, and w represents the width of the first feature). The first feature here can be the feature obtained by the last fully connected layer in the image processing model, or it can be the feature obtained by the last convolutional layer in the image processing model, and there is no limitation on this.

[0106] S803: Perform refinement processing according to the first feature of the target image to obtain a second feature of the target image.

[0107] The refinement processing here refers to the processing to improve the accuracy of the first feature of the target image; therefore, the accuracy of the second feature obtained by the refinement processing based on the first feature of the target image is higher than the accuracy of the first feature; the subsequent business processing based on the second feature with higher accuracy can further improve the accuracy of the business processing result. The specific implementation of step S803 can be referred to in conjunction with Fig. 9As shown, the following steps s21-s22 may be specifically included:

[0108] s21, performing feature aggregation processing according to the first feature of the target image to obtain the aggregated feature of the target image.

[0109] First, the first weight value of the first feature of the target image in the image space and the second weight value of the first feature in the image channel can be calculated; the image space here refers to the space where the height and width of the first feature are located. When calculating the first weight value, the formula shown in formula 1.3 can be used to calculate the weight value α of the first feature at any position (i, j) in all channels ij , and then determine the first weight value α of the first feature in the image space according to the weight values ​​of the first feature at all positions in all channels.

[0110]

[0111] In the above formula 1.3, is the value of the first feature at position (i, j) in the kth channel; c represents the number of channels of the first feature, h represents the height of the first feature, and w represents the width of the first feature; a and b are both constants, and their specific values ​​can be set according to empirical values, for example, set to 2.

[0112] When calculating the second weight value, the formula shown in formula 1.4 can be used to calculate the number Q of target positions that meet the conditions of the first feature in any channel (i.e., the kth channel): k , the so-called target position refers to the position where the value is greater than 0; where λ in formula 1.4 (ij) represents the value of any position (i, j) in the kth channel. Secondly, the weight value β of the first feature in the kth channel is calculated using the formula shown in formula 1.5 k ; Then, the second weight value β of the first feature on the image channel can be determined according to the weight value of the first feature in all channels.

[0113]

[0114]

[0115] After calculating the first weight value and the second weight value, weighted aggregation can be performed according to the first weight value, the second weight value and the first feature to obtain the aggregated features of the target image. Specifically, the first feature can be used to multiply the first weight value and the second weight value respectively to obtain the aggregated features of the target image. The embodiment of the present invention performs feature aggregation processing according to the first feature of the target image, which can enhance the feature weight of the main content of the target image, obtain the aggregated features that enhance the main content of the target image, and make the aggregated features more accurate. It should be noted that after obtaining the aggregated features of the target image, step s22 can be executed to obtain the second features of the target image; or the aggregated features of the target image can be directly used as the second features of the target image.

[0116] s22, performing dimensionality reduction processing on the aggregated features of the target image to obtain the second features of the target image.

[0117] As can be seen from the above, the first feature of the target image is c×h×w. Performing feature aggregation processing based on the first feature will not change the feature dimension, that is, the feature dimension of the aggregated features of the target image is still c×h×w. Therefore, to facilitate subsequent business processing, the embodiment of the present invention can perform dimensionality reduction processing on the aggregated features of the target image, reducing the feature dimension from c×h×w to c×1×1, thereby obtaining a second feature with a feature dimension of c×1×1. In the specific implementation process, the aggregated features of the target image can first be pooled to obtain the pooled features of the target image, and the feature dimension of the pooled features is c×1×1; the pooling processing here can include but is not limited to: sum pooling, maximum pooling, average pooling, and the like.

[0118] After obtaining the pooled features of the target image, the pooled features of the target image can be normalized to obtain the second features of the target image, so that the distribution of the second features of the target image is more uniform, so that the feature vector values ​​of the second image are maintained at a distribution with a mean of 0 and a variance of 1, thereby better performing subsequent business processing. It should be noted that, in another embodiment, after obtaining the pooled features of the target image, the pooled features of the target image can also be directly used as the second features of the target image.

[0119] S804: Execute the requested service according to the second feature of the target image.

[0120] As can be seen from the foregoing, the business request may include an image retrieval request, an image similarity calculation request, and the like. In one embodiment, the business request includes an image retrieval request; accordingly, the specific implementation of step S804 may be: first, a feature database for feature retrieval is obtained, the feature database includes multiple candidate images and image features of each candidate image; the image features of the candidate image may include the first feature of the candidate image or the second feature of the candidate image. Specifically, multiple candidate images may be obtained, and the image processing model may be called to extract features of each candidate image to obtain the first feature of each candidate image; each candidate image and the first feature of each candidate image are added to the feature database. Alternatively, after obtaining the first feature of each candidate image, a refinement process may be performed based on the first feature of each candidate image to obtain the second feature of each candidate image, and each candidate image and the second feature of each candidate image are added to the feature database. Secondly, similarity calculation may be performed based on the image features of each candidate image and the second feature of the target image to obtain the similarity between each candidate image and the target image; specifically, similarity calculation may be performed based on the image features of each candidate image and the second feature of the target image using calculation methods such as Euclidean distance and cosine similarity. Then, based on the similarity between each candidate image and the target image, the target candidate images that meet the preset conditions can be screened out from the feature database; the target candidate images are output; the preset conditions here may include but are not limited to: taking the candidate images whose similarity is greater than the similarity threshold as the target candidate images; or selecting a preset number of candidate images in order of similarity from high to low as the target candidate images. It should be noted that when the image retrieval request is a trademark retrieval request and the candidate images included in the feature database are trademark images, the feature database can be the Faiss database (an open source retrieval library provided by a third party), which includes multiple trademark images and a quick index of the trademark features (i.e., image features) of each trademark image to facilitate subsequent query retrieval.

[0121] In another embodiment, the business request includes an image similarity calculation request; accordingly, the specific implementation of step S804 may be: obtaining a reference image for similarity calculation with the target image; calling an image processing model to perform feature extraction on the reference image to obtain a first feature of the reference image; and performing refinement processing based on the first feature of the reference image to obtain a second feature of the reference image; then performing similarity calculation based on the second feature of the target image and the second feature of the reference image to obtain the similarity between the target image and the reference image; and outputting the similarity between the target image and the reference image.

[0122] In the embodiment of the present invention, since the image processing model is adopted Figure 2 or Figure 6The model training method shown is trained, and Figure 2 or Figure 6 The model training method shown can effectively convert the unsupervised model training task into a supervised model training task, so that the image processing model can better learn the image features in the sample image, thereby improving the accuracy of the image processing model and the feature extraction capability; therefore, the use of the image processing model for feature extraction can improve the accuracy of the first feature. After obtaining the first feature, a refined processing is performed based on the first feature to obtain a second feature with higher accuracy, so that the business processing result obtained by performing business processing based on the second feature is more accurate.

[0123] The following uses the image processing model in the trademark search scenario as an example to illustrate the business processing method proposed in the embodiment of the present invention. The specific implementation methods of other application scenarios can be found in the embodiment of the present invention, which will not be repeated here. Fig.10a As shown: the terminal can pre-call the trained image processing model to build a feature database, which includes multiple candidate trademark images (hereinafter referred to as candidate trademarks) and trademark features of each candidate trademark (i.e., image features); the trademark features of the candidate trademark can be obtained by calling the image processing model to extract features from the candidate trademark, or the trademark features of the candidate trademark can be obtained by calling the image processing model to extract features from the candidate trademark and then refining the extracted features. When a user wants to retrieve a target trademark image (hereinafter referred to as the target trademark), the terminal can use the terminal to open the trademark search system; after the terminal detects the opening operation for the trademark search system, it can run the trademark search system and provide the user with a user interface for uploading the target trademark. The user can click the upload button 1 in the user interface to enter the upload interface to upload the target trademark, as shown in Fig.10b After the user uploads the target trademark, he can click the confirmation button 2 in the upload interface to trigger the terminal to obtain the target trademark, such as Fig.10c As shown. After acquiring the target trademark, the terminal can call the image processing model to extract the features of the target trademark and obtain the image features of the target trademark; the image features of the target trademark may include the first feature of the target trademark or the second feature of the target trademark. Then, the target features of the target trademark can be matched with the trademark features of each candidate trademark stored in the feature database to obtain multiple (for example, 6) target candidate trademarks that may be homologous trademark images with the target trademark, and output the target candidate trademark, as shown in FIG. Fig.10c shown.

[0124] In order to better illustrate the beneficial effects of the business processing method proposed in the embodiment of the present invention, the inventor also used the business processing method of the embodiment of the present invention and the existing trademark search method to conduct a trademark search test on the same target trademark. The search results obtained by searching the target trademark using the business processing method in the embodiment of the present invention can be found in Fig.10c As shown; the search results obtained by searching the target trademark using the existing trademark search method can be found in Fig.10d As shown. Fig.10c The search results shown are Fig.10d The search results shown show that Fig.10d There are many trademarks that are not similar to the target trademark in the search results shown. Fig.10c The similarity between each target candidate trademark in the search results shown and the input target trademark is relatively high. It can be seen that the business processing method adopted in the embodiment of the present invention for trademark search can greatly improve the accuracy of homologous trademark search, and quickly and accurately return the search trademark results expected by the user. It can be seen that the business processing method proposed in the embodiment of the present invention can accurately retrieve the homologous trademark corresponding to the target trademark, and completely solve the problems of low accuracy in homologous trademark search and a large number of duplicate registered trademarks and invalid trademarks in trademark applications. Applying this business processing method in a trademark search system (or trademark registration system) can greatly reduce the human resources involved in the trademark registration and trademark review process, and improve the human and time costs of the State Trademark Office and enterprises in trademark review and trademark registration.

[0125] Based on the description of the above model training method embodiment, the present invention also discloses a model training device; the model training device can be a computer program (including program code) running in a terminal. Figure 2 or Figure 6 See the method shown in Fig.11 , the model training device can run the following units:

[0126] An acquisition unit 101 is used to acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images;

[0127] A construction unit 102 is used to construct a data structure based on the image relationship and the multiple sample images; wherein the data structure is composed of multiple elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated element pair stores the sample images with the association relationship;

[0128] A clustering unit 103, used to traverse each element of the data structure and cluster the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same;

[0129] The training unit 104 is used to perform model training on the image processing model using the multiple sample images and the category information of each sample image.

[0130] In one implementation, when the construction unit 102 is used to construct a data structure based on the image relationship and the plurality of sample images, it may be specifically used to:

[0131] Using a plurality of elements to store the plurality of sample images, one element storing one sample image;

[0132] Determining an associated element pair from the plurality of elements based on the image relationship, wherein the associated element pair has the association relationship between sample images stored therein;

[0133] The associated pairs of elements are connected to obtain a data structure.

[0134] In another embodiment, the data structure includes an undirected graph, and the elements of the data structure are nodes of the undirected graph; accordingly, when the clustering unit 103 is used to traverse each element of the data structure and cluster the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set, it can be specifically used to:

[0135] Traversing each node of the undirected graph based on a depth restriction condition to obtain multiple connected graphs;

[0136] The sample images stored in the nodes of the same connected graph are added to the same cluster set, and the sample images in the same cluster set are labeled with the same category information.

[0137] In another embodiment, when the clustering unit 103 is used to traverse each node of the undirected graph based on the depth restriction condition to obtain multiple connected graphs, it can be specifically used to:

[0138] Select any unvisited node from the undirected graph as the starting node of the current traversal process, and visit the starting node;

[0139] Based on the node width-first traversal principle and along the traversal direction from shallow to deep node depth, the remaining nodes other than the starting node and not visited are visited in the undirected graph; the node width-first traversal principle refers to the principle of preferentially traversing nodes at the same node depth;

[0140] If the node depth of the current node being visited is greater than a preset threshold, the current traversal process is terminated, and a connected graph is constructed using the nodes visited in the current traversal process; wherein the node depth of the current node is determined according to the number of edges between the current node and the starting node;

[0141] The above steps are repeated until all nodes of the undirected graph are visited, thereby obtaining multiple connected graphs.

[0142] In another embodiment, when the training unit 104 is used to perform model training on the image processing model using the multiple sample images and the category information of each sample image, it can be specifically used to:

[0143] Calling the image processing model to perform feature extraction on each sample image to obtain image features of each sample image;

[0144] Calling the image processing model to perform category prediction according to the image features of each sample image to obtain prediction information of each sample image;

[0145] Model training is performed on the image processing model according to the prediction information of each sample image and the category information of each sample image.

[0146] In another embodiment, when the training unit 104 is used to perform model training on the image processing model according to the prediction information of each sample image and the category information of each sample image, it can be specifically used to:

[0147] Obtaining a loss function of the image processing model, wherein the loss function includes: a feature distance loss function and a feature classification loss function;

[0148] Calculating the value of the loss function according to the prediction information of each sample image and the category information of each sample image;

[0149] The image processing model is optimized according to the principle of reducing the value of the loss function.

[0150] According to one embodiment of the present invention, Figure 2 or Figure 6 Each step involved in the method shown can be performed by Fig.11 The model training device shown is executed by each unit. For example, Figure 2 The steps S201-S204 shown in FIG. 1 can be respectively Fig.11 The acquisition unit 101, the construction unit 102, the clustering unit 103 and the training unit 104 shown in are executed; for example, Figure 6 The steps S601-S603 shown in FIG. 1 can be respectively Fig.11The acquisition unit 101, the construction unit 102 and the clustering unit 103 shown in FIG. 1 are executed, and steps S604-S606 can be performed by Fig.11 The training unit 104 shown in FIG.

[0151] According to another embodiment of the present invention, Fig.11 The various units in the model training device shown can be separately or completely combined into one or several other units to form, or some of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present invention, other units can also be included based on the model training device. In practical applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present invention, it can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 2 or Figure 6 A computer program (including program code) for each step involved in the corresponding method shown in Fig.11 The model training device shown in the embodiment of the present invention is used to implement the model training method of the embodiment of the present invention. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0152] In the process of model training, the embodiment of the present invention can first obtain a sample set, and construct a data structure based on image relationships and multiple sample images. Since any two connected elements in the data structure are associated element pairs, there is an association relationship between the sample images stored in the associated element pairs, so each element of the data structure can be traversed and the sample images stored in each element can be clustered to obtain multiple cluster sets and category information of sample images in each cluster set; the category information of the sample images is obtained by using the data structure, and no manual labeling is required by the labeling personnel, which can improve the efficiency of model training and reduce labor costs. After obtaining the category information of each sample image, multiple sample images and the category information of each sample image can be used to train the image processing model; it can be seen that the embodiment of the present invention can also effectively convert the unsupervised model training task into a supervised model training task, so that the image processing model can better learn the image features in the sample image, thereby improving the accuracy of the image processing model; and by constructing a sample image training image processing model with category information, the feature expression ability of the image processing model for simple texture images can also be improved.

[0153] Based on the description of the above-mentioned business processing method embodiment, the embodiment of the present invention further discloses a business processing device; the business processing device may be a computer program (including program code) running in a terminal. The business processing device may execute Figure 8 See the method shown in Fig.12 , the service processing device may run the following units:

[0154] The acquisition unit 201 is used to acquire a target image to be processed in response to a service request;

[0155] The processing unit 202 is used to call the image processing model to extract the features of the target image to obtain the first feature of the target image; the image processing model is based on Figure 2 or Figure 6 The model training method shown is trained;

[0156] The processing unit 202 is used to perform refinement processing according to the first feature of the target image to obtain the second feature of the target image;

[0157] The service unit 203 is configured to execute the requested service according to the second feature of the target image.

[0158] In one implementation, when the processing unit 202 is used to perform refinement processing according to the first feature of the target image to obtain the second feature of the target image, it can be specifically used to:

[0159] Performing feature aggregation processing according to the first feature of the target image to obtain an aggregated feature of the target image;

[0160] Performing dimensionality reduction processing on the aggregated features of the target image to obtain a second feature of the target image.

[0161] In yet another embodiment, when the processing unit 202 is used to perform feature aggregation processing according to the first feature of the target image to obtain the aggregated feature of the target image, it can be specifically used to:

[0162] Calculating a first weight value of a first feature of the target image in an image space and a second weight value of the first feature in an image channel;

[0163] Weighted aggregation is performed according to the first weight value, the second weight value, and the first feature to obtain an aggregated feature of the target image.

[0164] In another embodiment, when the processing unit 202 is used to perform dimensionality reduction processing on the aggregated features of the target image to obtain the second features of the target image, it can be specifically used to:

[0165] Performing pooling processing on the aggregated features of the target image to obtain pooled features of the target image;

[0166] Normalize the pooled features of the target image to obtain a second feature of the target image.

[0167] In another embodiment, the service request includes an image retrieval request; accordingly, when the service unit 203 is used to execute the requested service according to the second feature of the target image, it can be specifically used to:

[0168] Acquire a feature database for feature retrieval, wherein the feature database includes a plurality of candidate images and image features of each candidate image;

[0169] Calculate similarity based on the image features of each candidate image and the second feature of the target image to obtain the similarity between each candidate image and the target image;

[0170] Based on the similarity between each candidate image and the target image, screening out a target candidate image that meets a preset condition from the feature database;

[0171] The target candidate image is output.

[0172] According to one embodiment of the present invention, Figure 8 Each step involved in the method shown can be performed by Fig.12 The business processing device shown in FIG. 1 is executed by each unit in the business processing device shown in FIG. Figure 8 The step S801 shown in FIG. Fig.12 The acquisition unit 201 shown in FIG. 1 is executed, and steps S802-S803 can be performed by Fig.12 The processing unit 202 shown in FIG. 1 is executed, and step S804 can be performed by Fig.12 According to another embodiment of the present invention, Fig.12 The various units in the business processing device shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present invention, other units can also be included based on the business processing device. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present invention, it is possible to execute the following by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 8 A computer program (including program code) for each step involved in the corresponding method shown in Fig.12 The business processing device shown in the embodiment of the present invention is used to implement the business processing method of the embodiment of the present invention. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0173] In the embodiment of the present invention, since the image processing model is adopted Figure 2 or Figure 6 The model training method shown is trained, and Figure 2 or Figure 6 The model training method shown can effectively convert the unsupervised model training task into a supervised model training task, so that the image processing model can better learn the image features in the sample image, thereby improving the accuracy of the image processing model and the feature extraction capability; therefore, the use of the image processing model for feature extraction can improve the accuracy of the first feature. After obtaining the first feature, a refined processing is performed based on the first feature to obtain a second feature with higher accuracy, so that the business processing result obtained by performing business processing based on the second feature is more accurate.

[0174] Based on the description of the above method embodiment and device embodiment, the embodiment of the present invention further provides a terminal. Fig.13, the terminal at least includes a processor 301, an input device 302, an output device 303 and a computer storage medium 304. Among them, the processor 301, the input device 302, the output device 303 and the computer storage medium 304 can be connected via a bus or other means. The computer storage medium 304 can be stored in the memory of the terminal, and the computer storage medium 304 is used to store a computer program, and the computer program includes program instructions. The processor 301 is used to execute the program instructions stored in the computer storage medium 304. The processor 301 (or CPU (Central Processing Unit)) is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function. Optionally, the terminal may also include an image processing model.

[0175] In one embodiment, the processor 301 described in the embodiment of the present invention can be used to perform a series of training processes on the image processing model, including: obtaining a sample set, the sample set including multiple sample images and image relationships, the image relationship including the association relationship between at least two sample images; constructing a data structure based on the image relationship and the multiple sample images; wherein the data structure is composed of multiple elements, one element stores a sample image, any two connected elements are associated element pairs, and the associated element pairs store sample images with the association relationship; traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same; using the multiple sample images and the category information of each sample image to perform model training on the image processing model, etc. In another embodiment, the processor 301 described in the embodiment of the present invention can also be used to perform a series of business processes, including: in response to a business request, obtaining a target image to be processed; calling the image processing model to extract features from the target image to obtain the first feature of the target image; the image processing model is Figure 2 or Figure 6 The model is obtained by training using the model training method shown; a first feature of the target image is refined to obtain a second feature of the target image; the requested service is executed according to the second feature of the target image, and so on.

[0176] The embodiment of the present invention also provides a computer storage medium (Memory), which is a memory device in the terminal for storing programs and data. It is understandable that the computer storage medium here can include both the built-in storage medium in the terminal and the extended storage medium supported by the terminal. The computer storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor 301 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0177] In one embodiment, the processor 301 may load and execute one or more first instructions stored in a computer storage medium to implement the corresponding steps of the method in the above-mentioned model training embodiment; in a specific implementation, the processor 301 loads one or more first instructions in the computer storage medium and executes the following steps:

[0178] Acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images;

[0179] Constructing a data structure based on the image relationship and the multiple sample images; wherein the data structure is composed of multiple elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated relationship exists between the sample images stored in the associated element pair;

[0180] Traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same;

[0181] The image processing model is trained using the multiple sample images and the category information of each sample image.

[0182] In one implementation, when constructing a data structure based on the image relationship and the plurality of sample images, the one or more first instructions are loaded and executed by the processor 301:

[0183] Using a plurality of elements to store the plurality of sample images, one element storing one sample image;

[0184] Determining an associated element pair from the plurality of elements based on the image relationship, wherein the associated element pair has the association relationship between sample images stored therein;

[0185] The associated pairs of elements are connected to obtain a data structure.

[0186] In another embodiment, the data structure includes an undirected graph, and the elements of the data structure are nodes of the undirected graph; accordingly, when traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set, the one or more first instructions are loaded and executed by the processor 301:

[0187] Traversing each node of the undirected graph based on a depth restriction condition to obtain multiple connected graphs;

[0188] The sample images stored in the nodes of the same connected graph are added to the same cluster set, and the sample images in the same cluster set are labeled with the same category information.

[0189] In another implementation, when traversing each node of the undirected graph based on the depth restriction condition to obtain multiple connected graphs, the one or more first instructions are loaded and executed by the processor 301:

[0190] Select any unvisited node from the undirected graph as the starting node of the current traversal process, and visit the starting node;

[0191] Based on the node width-first traversal principle and along the traversal direction from shallow to deep node depth, the remaining nodes other than the starting node and not visited are visited in the undirected graph; the node width-first traversal principle refers to the principle of preferentially traversing nodes at the same node depth;

[0192] If the node depth of the current node being visited is greater than a preset threshold, the current traversal process is terminated, and a connected graph is constructed using the nodes visited in the current traversal process; wherein the node depth of the current node is determined according to the number of edges between the current node and the starting node;

[0193] The above steps are repeated until all nodes of the undirected graph are visited, thereby obtaining multiple connected graphs.

[0194] In another embodiment, when the image processing model is trained using the multiple sample images and the category information of each sample image, the one or more first instructions are loaded and executed by the processor 301:

[0195] Calling the image processing model to perform feature extraction on each sample image to obtain image features of each sample image;

[0196] Calling the image processing model to perform category prediction according to the image features of each sample image to obtain prediction information of each sample image;

[0197] Model training is performed on the image processing model according to the prediction information of each sample image and the category information of each sample image.

[0198] In another embodiment, when the image processing model is trained according to the prediction information of each sample image and the category information of each sample image, the one or more first instructions are loaded and executed by the processor 301:

[0199] Obtaining a loss function of the image processing model, wherein the loss function includes: a feature distance loss function and a feature classification loss function;

[0200] Calculating the value of the loss function according to the prediction information of each sample image and the category information of each sample image;

[0201] The image processing model is optimized according to the principle of reducing the value of the loss function.

[0202] In another embodiment, the processor 301 may load and execute one or more second instructions stored in a computer storage medium to implement the corresponding steps of the method in the above-mentioned business processing embodiment; in a specific implementation, the processor 301 loads one or more second instructions in the computer storage medium and executes the following steps:

[0203] In response to a service request, obtaining a target image to be processed;

[0204] Calling an image processing model to extract features of the target image to obtain a first feature of the target image; the image processing model is adopted Figure 2 or Figure 6 The model training method shown is trained;

[0205] Performing refinement processing according to the first feature of the target image to obtain a second feature of the target image;

[0206] The requested service is executed according to the second feature of the target image.

[0207] In one implementation, when performing refinement processing according to the first feature of the target image to obtain the second feature of the target image, the one or more second instructions are loaded and executed by the processor 301:

[0208] Performing feature aggregation processing according to the first feature of the target image to obtain an aggregated feature of the target image;

[0209] Performing dimensionality reduction processing on the aggregated features of the target image to obtain a second feature of the target image.

[0210] In another implementation manner, when performing feature aggregation processing according to the first feature of the target image to obtain the aggregated feature of the target image, the one or more second instructions are loaded and executed by the processor 301:

[0211] Calculating a first weight value of a first feature of the target image in an image space and a second weight value of the first feature in an image channel;

[0212] Weighted aggregation is performed according to the first weight value, the second weight value, and the first feature to obtain an aggregated feature of the target image.

[0213] In another implementation manner, when performing dimensionality reduction processing on the aggregated features of the target image to obtain the second features of the target image, the one or more second instructions are loaded and executed by the processor 301:

[0214] Performing pooling processing on the aggregated features of the target image to obtain pooled features of the target image;

[0215] Normalize the pooled features of the target image to obtain a second feature of the target image.

[0216] In another embodiment, the service request includes an image retrieval request; accordingly, when executing the requested service according to the second feature of the target image, the one or more second instructions are loaded and executed by the processor 301:

[0217] Acquire a feature database for feature retrieval, wherein the feature database includes a plurality of candidate images and image features of each candidate image;

[0218] Calculate similarity based on the image features of each candidate image and the second feature of the target image to obtain the similarity between each candidate image and the target image;

[0219] Based on the similarity between each candidate image and the target image, screening out a target candidate image that meets a preset condition from the feature database;

[0220] The target candidate image is output.

[0221] In the process of model training, the embodiment of the present invention can first obtain a sample set, and construct a data structure based on image relationships and multiple sample images. Since any two connected elements in the data structure are associated element pairs, there is an association relationship between the sample images stored in the associated element pairs, so each element of the data structure can be traversed and the sample images stored in each element can be clustered to obtain multiple cluster sets and category information of sample images in each cluster set; the category information of the sample images is obtained by using the data structure, and no manual labeling is required by the labeling personnel, which can improve the efficiency of model training and reduce labor costs. After obtaining the category information of each sample image, multiple sample images and the category information of each sample image can be used to train the image processing model; it can be seen that the embodiment of the present invention can also effectively convert the unsupervised model training task into a supervised model training task, so that the image processing model can better learn the image features in the sample image, thereby improving the accuracy of the image processing model; and by constructing a sample image training image processing model with category information, the feature expression ability of the image processing model for simple texture images can also be improved.

[0222] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A model training method, characterized in that: include: Acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images; The association relationship between the two sample images refers to a relationship used to reflect the similarity between the two sample images; Constructing a data structure based on the image relationship and the multiple sample images; wherein the data structure is composed of multiple elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated relationship exists between the sample images stored in the associated element pair; Based on the depth restriction condition, each element of the data structure is traversed and the sample images stored in each element are clustered to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same; the depth restriction condition includes terminating the current traversal process if the element depth of the current element being accessed in the current traversal process is greater than a preset threshold; the element depth of the current element is determined according to the number of edges between the current element and the starting element in the current traversal process; The image processing model is trained using the multiple sample images and the category information of each sample image.

2. The method according to claim 1, characterized in that The constructing a data structure based on the image relationship and the plurality of sample images comprises: Using a plurality of elements to store the plurality of sample images, one element storing one sample image; determining associated pairs of elements from the plurality of elements based on the image relationship; The associated pairs of elements are connected to obtain a data structure.

3. The method according to claim 1, characterized in that The data structure comprises an undirected graph, and the elements of the data structure are nodes of the undirected graph; The traversing each element of the data structure and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set includes: Traversing each node of the undirected graph based on a depth restriction condition to obtain multiple connected graphs; The sample images stored in the nodes of the same connected graph are added to the same cluster set, and the sample images in the same cluster set are labeled with the same category information.

4. The method according to claim 3, characterized in that The step of traversing each node of the undirected graph based on the depth restriction condition to obtain multiple connected graphs includes: Select any unvisited node from the undirected graph as the starting node of the current traversal process, and visit the starting node; Based on the node width-first traversal principle and along the traversal direction from shallow to deep node depth, the remaining nodes other than the starting node and not visited are visited in the undirected graph; the node width-first traversal principle refers to the principle of preferentially traversing nodes at the same node depth; If the node depth of the current node being visited is greater than a preset threshold, the current traversal process is terminated, and a connected graph is constructed using the nodes visited in the current traversal process; wherein the node depth of the current node is determined according to the number of edges between the current node and the starting node; The above steps are repeated until all nodes of the undirected graph are visited, thereby obtaining multiple connected graphs.

5. The method according to any one of claims 1 to 4, characterized in that: The method of using the plurality of sample images and the category information of each sample image to perform model training on the image processing model includes: Calling the image processing model to perform feature extraction on each sample image to obtain image features of each sample image; Calling the image processing model to perform category prediction according to the image features of each sample image to obtain prediction information of each sample image; Model training is performed on the image processing model according to the prediction information of each sample image and the category information of each sample image.

6. The method according to claim 5, characterized in that The performing model training on the image processing model according to the prediction information of each sample image and the category information of each sample image comprises: Obtaining a loss function of the image processing model, wherein the loss function includes: a feature distance loss function and a feature classification loss function; Calculating the value of the loss function according to the prediction information of each sample image and the category information of each sample image; The image processing model is optimized according to the principle of reducing the value of the loss function.

7. A business processing method, characterized in that: include: In response to a service request, obtaining a target image to be processed; Calling an image processing model to extract features of the target image to obtain a first feature of the target image; the image processing model is trained using the model training method according to any one of claims 1 to 6; Performing refinement processing according to the first feature of the target image to obtain a second feature of the target image; The requested service is executed according to the second feature of the target image.

8. The method according to claim 7, characterized in that The performing refinement processing according to the first feature of the target image to obtain the second feature of the target image includes: Performing feature aggregation processing according to the first feature of the target image to obtain an aggregated feature of the target image; Performing dimensionality reduction processing on the aggregated features of the target image to obtain a second feature of the target image.

9. The method according to claim 8, characterized in that The performing feature aggregation processing according to the first feature of the target image to obtain the aggregated feature of the target image includes: Calculating a first weight value of a first feature of the target image in an image space and a second weight value of the first feature in an image channel; Weighted aggregation is performed according to the first weight value, the second weight value, and the first feature to obtain an aggregated feature of the target image.

10. The method according to claim 8, characterized in that The performing dimensionality reduction processing on the aggregated features of the target image to obtain the second features of the target image includes: Performing pooling processing on the aggregated features of the target image to obtain pooled features of the target image; Normalize the pooled features of the target image to obtain a second feature of the target image.

11. The method according to any one of claims 7 to 10, characterized in that: The service request includes an image retrieval request, and the executing the requested service according to the second feature of the target image includes: Acquire a feature database for feature retrieval, wherein the feature database includes a plurality of candidate images and image features of each candidate image; Performing similarity calculation based on the image features of each candidate image and the second feature of the target image to obtain the similarity between each candidate image and the target image; Based on the similarity between each candidate image and the target image, screening out a target candidate image that meets a preset condition from the feature database; The target candidate image is output.

12. A model training device, characterized in that: include: an acquisition unit, configured to acquire a sample set, wherein the sample set includes a plurality of sample images and image relationships, wherein the image relationship includes an association relationship between at least two sample images; the association relationship between the two sample images refers to a relationship used to reflect the similarity between the two sample images; A construction unit, configured to construct a data structure based on the image relationship and the plurality of sample images; wherein the data structure is composed of a plurality of elements, one element stores one sample image, any two connected elements are an associated element pair, and the associated relationship exists between the sample images stored in the associated element pair; A clustering unit, used for traversing each element of the data structure based on a depth restriction condition and clustering the sample images stored in each element to obtain multiple cluster sets and category information of the sample images in each cluster set; wherein the category information of the sample images in each cluster set is the same; the depth restriction condition includes terminating the current traversal process if the element depth of the current element being accessed in the current traversal process is greater than a preset threshold; the element depth of the current element is determined according to the number of edges between the current element and the starting element in the current traversal process; The training unit is used to use the multiple sample images and the category information of each sample image to perform model training on the image processing model.

13. A service processing device, characterized in that: include: An acquisition unit, used for acquiring a target image to be processed in response to a service request; A processing unit, configured to call an image processing model to perform feature extraction on the target image to obtain a first feature of the target image; the image processing model is trained using the model training method according to any one of claims 1 to 6; The processing unit is used to perform refinement processing according to the first feature of the target image to obtain the second feature of the target image; A service unit is used to execute the requested service according to the second feature of the target image.

14. A terminal comprising an input device and an output device, characterized in that: Also includes: a processor adapted to implement one or more instructions; as well as, A computer storage medium storing one or more first instructions, wherein the one or more first instructions are suitable for being loaded by the processor and executing the model training method as described in any one of claims 1 to 6; or, the computer storage medium storing one or more second instructions, wherein the one or more second instructions are suitable for being loaded by the processor and executing the business processing method as described in any one of claims 7 to 11.

15. A computer storage medium, characterized in that: The computer storage medium stores one or more first instructions, and the one or more first instructions are suitable for being loaded by the processor and executing the model training method as described in any one of claims 1-6; or, the computer storage medium stores one or more second instructions, and the one or more second instructions are suitable for being loaded by the processor and executing the business processing method as described in any one of claims 7-11.

Images