Model training method and device and chart identification method

Abstract

The invention discloses a model training method and device and a chart recognition method, and relates to the technical field of model training, and the method comprises the steps: carrying out the feature extraction of each training image in a pre-obtained training set, and obtaining an initial feature vector corresponding to each training image; obtaining a first classification feature vector of each initial feature vector; performing disturbance operation on each initial feature vector to obtain a disturbed vector corresponding to each initial feature vector; obtaining a second classification feature vector of each disturbed vector; and training the initial model based on a first difference between the first classification feature vector and the corresponding standard chart type and a second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector. The model is trained through the first difference and the second difference, so that the robustness of the trained model and the accuracy of chart type identification are improved.

Description

Technical Field

[0001] The present application relates to the field of model training technology, and more specifically, to a model training method, device and chart recognition method. Background Art

[0002] At present, with the development of computer technology, electronic devices can be used to identify images and determine the type of charts in the images. However, the current methods for identifying the type of charts in images have low robustness and accuracy. Summary of the invention

[0003] The present application proposes a model training method, device and chart recognition method.

[0004] In a first aspect, an embodiment of the present application provides a model training method, comprising: performing feature extraction on each training image in a pre-acquired training set to obtain an initial feature vector corresponding to each training image; obtaining a first classification feature vector for each of the initial feature vectors; performing a perturbation operation on each of the initial feature vectors to obtain a perturbed vector corresponding to each of the initial feature vectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial feature vector; obtaining a second classification feature vector for each of the perturbed vectors; and training an initial model based on a first difference between the first classification feature vector and the corresponding standard chart type, and a second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to a target range.

[0005] In the second aspect, the embodiment of the present application also provides a model training device, including: an extraction unit, a first feature vector acquisition unit, a perturbation unit, a second feature vector acquisition unit and a training unit. The extraction unit is used to extract features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image; the first feature vector acquisition unit is used to obtain a first classification feature vector for each of the initial feature vectors; the perturbation unit is used to perform a perturbation operation on each of the initial feature vectors to obtain a perturbed vector corresponding to each of the initial feature vectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial feature vector; the second feature vector acquisition unit is used to obtain a second classification feature vector for each of the perturbed vectors; the training unit is used to train the initial model based on the first difference between the first classification feature vector and the corresponding standard chart type, and the second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to the target range.

[0006] In a third aspect, an embodiment of the present application further provides a chart recognition method, the method comprising obtaining an image to be recognized; using a chart recognition model to recognize the image to be recognized, and outputting a chart type corresponding to the image to be recognized, wherein the chart recognition model is trained based on the model training method described in the first aspect.

[0007] The model training method, device and chart recognition method provided by the embodiment of the present application first extract features from each training image in the pre-acquired training set to obtain the initial feature vector corresponding to each training image; then obtain the first classification feature vector of each of the initial feature vectors; perform a perturbation operation on each of the initial feature vectors to obtain the perturbed vector corresponding to each of the initial feature vectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial feature vector; then obtain the second classification feature vector of each of the perturbed vectors; thereby training the initial model based on the first difference between the first classification feature vector and the corresponding standard chart type, and the second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to the target range. In the process of identifying the chart type of an image, the image may be subject to some disturbances, so that the perturbed image may be mistakenly identified as other chart types, that is, the chart type is incorrectly identified, and the robustness is low. In the present application, by performing a perturbation operation on the initial feature vector and then training the initial model based on the first difference and the second difference, the initial model can overcome the impact of the perturbation operation and more accurately learn the features in each training image, thereby improving the robustness of the trained initial model during use and the accuracy of chart recognition.

[0008] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1An application scenario diagram of the model training method provided in an embodiment of the present application is shown;

[0011] Figure 2 A method flow chart of the model training method provided in an embodiment of the present application is shown;

[0012] Figure 3 A method flow chart of a model training method provided by another embodiment of the present application is shown;

[0013] Figure 4 A method flow chart of a model training method provided by yet another embodiment of the present application is shown;

[0014] Figure 5 A flow chart of a method for fine-tuning a model to be verified provided in an embodiment of the present application is shown;

[0015] Figure 6 A method flow chart of a model training method provided by another embodiment of the present application is shown;

[0016] Figure 7 A method flow chart of a model training method provided by another embodiment of the present application is shown;

[0017] Figure 8 A method flow chart of a diagram recognition method provided by an embodiment of the present application is shown;

[0018] Fig. 9 A structural block diagram of a model training device provided in an embodiment of the present application is shown;

[0019] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0020] Fig.11 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0022] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0023] At present, with the development of computer technology, electronic devices can be used to identify images and determine the type of chart in the image. However, the current methods for identifying the type of chart in an image have low robustness and accuracy. How to improve the robustness and accuracy of identifying the type of chart in an image is an urgent problem to be solved.

[0024] Currently, the chart type of the input image can be detected to determine the chart type of the image.

[0025] However, the inventors found in their research that in the process of identifying the chart type of an image, the image may be disturbed. Therefore, the current recognition method may mistakenly identify the disturbed image as other chart types, that is, the chart type is incorrectly identified, and the robustness is low.

[0026] Therefore, in order to solve or partially solve the above problems, the present application provides a model training method, device and chart recognition method.

[0027] See also Figure 1 , Figure 1 An application scenario diagram of the model training method provided in an embodiment of the present application is shown, namely, a model training scenario 100, which may include an electronic device 110.

[0028] The electronic device 110 can access the Internet by, for example, accessing the Internet through wireless communication technologies such as Wi-Fi and Bluetooth; the electronic device 110 can also access the Internet through a wired method, for example, accessing the Internet through an Rj45 network cable or optical fiber.

[0029] The user can control the electronic device 110 so that the electronic device executes the model training method. For example, the user can directly operate the electronic device 110 to control the electronic device to execute the model training method, wherein the electronic device 110 can be locally deployed with an initial model for training. For a detailed description, please refer to the subsequent implementation. The user can also control the electronic device 110 through another electronic device that is also connected to the Internet, thereby controlling the electronic device 110 to execute the training method.

[0030] See also Figure 2 , Figure 2A method flow chart of a model training method provided in an embodiment of the present application is shown. The model training method can be applied to Figure 1 The electronic device in the video generation scene shown can specifically use the processor of the electronic device as the execution subject of the model training method. The model training method can include steps S110 to S150.

[0031] Step S110: extracting features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image.

[0032] In some embodiments, the model can be used to identify the chart type of the image, thereby determining the chart type to which the image belongs. Exemplary, chart types may include mind maps, bar charts, pie charts, and tables, etc. And identifying the chart type of the image through the model may include open set recognition (OSR) and closed set recognition (OSR). Among them, closed set recognition means that during the training and testing process, all possible categories are known, that is, the categories in the test set are completely included in the categories of the training set. The task of the model is to classify the input data into these known categories. In open set recognition, the model not only needs to correctly classify data of known categories, but also needs to be able to recognize data of unknown categories.

[0033] Taking chart types including mind maps, bar charts, pie charts and tables as an example to illustrate open set recognition, the model is used to identify the chart type of the image, so as to determine whether the image belongs to the mind map, bar chart, pie chart or table type, or if it does not belong to any of the above types, then it can be determined that the image belongs to a non-chart type.

[0034] Therefore, in the embodiment provided in the present application, the initial model can be trained by a training set to obtain a trained initial model, and the trained initial model can have the ability to perform open set recognition on the chart type of the image.

[0035] The training set may include multiple training images, and the training images may include images of different chart types. It is understandable that each chart type may correspond to chart content, for example, a mind map may correspond to each node as chart content; a column chart may correspond to a column area and related text information as chart content; a pie chart may correspond to a sector area as chart content; and a table may correspond to table data as chart content.

[0036] Therefore, for some embodiments, an image including a chart content corresponding to a chart may be obtained as a training image. Exemplarily, a screenshot operation may be performed on the chart content corresponding to a mind map, a bar chart, a pie chart, or a table displayed on an electronic device, thereby obtaining an initial image corresponding to the mind map, a bar chart, a pie chart, or a table. Specifically, for example, a screenshot tool may be used to capture the chart content displayed on the screen.

[0037] Optionally, another electronic device may take a screenshot of the chart content displayed on another electronic device to obtain an initial image corresponding to the chart content, and then the other electronic device sends the initial image obtained by the screenshot operation to the electronic device.

[0038] In another exemplary embodiment, the chart generation application can generate charts of different chart types. Therefore, the chart generation application can also be used to export the generated charts of different types into image formats, thereby obtaining the initial image.

[0039] In another exemplary embodiment, the images of various chart types can be directly searched and obtained from the Internet by an electronic device as the initial images. For example, the images of various chart types can be directly downloaded.

[0040] As another example, the electronic device is generally also configured with an image acquisition module, so that charts of various chart types can be photographed by the image acquisition module of the electronic device to generate corresponding images as initial images.

[0041] The initial images obtained through the above examples can be directly used as training images in the training set. Optionally, the training set can also be expanded according to the initial images obtained above, so as to obtain a training set including more training images. The detailed method can be referred to the introduction of the subsequent embodiments.

[0042] The training images in the training set may also correspond to standard chart types. Thus, the initial model may be trained by the training images in the training set. The initial model may be a neural network model, which may be a pre-trained model obtained through the network, or a model obtained by fine-tuning the obtained pre-trained model, which is not specifically limited in the embodiments of the present application.

[0043] Therefore, the initial model can be used to extract features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image.

[0044] Exemplarily, a feature extraction network (Feature Extractor) may be used to extract features from training images, thereby obtaining an initial feature vector corresponding to each training image.

[0045] In some embodiments, a tensor representation of a training image may be first obtained, and then the training image represented by the tensor may be input into a feature extraction network, thereby obtaining an output of the feature extraction network as an initial feature vector corresponding to each training image.

[0046] The feature extraction network may be a MobileNet V2 network. By using a linear layer with SpectralNorm as the projection layer of the feature extraction network, the severity of numerical changes can be effectively limited, making the gradient during weight update more stable during subsequent training.

[0047] Optionally, the output of the feature extraction network can also be used as an input to a feature embedding representation network, so as to obtain an initial feature vector corresponding to each training image through the feature embedding representation network.

[0048] Step S120: Obtain a first classification feature vector of each of the initial feature vectors.

[0049] Then, the first classification feature vector of each of the initial feature vectors can be obtained. In the embodiment provided in the present application, the first classification feature vector of each of the initial feature vectors can be obtained through the initial model. That is, the initial model can be used to perform a chart type recognition operation on the initial feature vector of each training image, thereby determining the first classification feature vector of each of the initial feature vectors. The first classification feature vector can be used to characterize the probability of each chart type to which the corresponding training image belongs.

[0050] Exemplarily, the chart types include the mind map, bar chart, pie chart and table shown above. The first classification feature vector can be expressed as [0.3, 0.2, 0.25, 0.1], where 0.3 is used to characterize the probability that the training image is a mind map chart type; 0.2 is used to characterize the probability that the training image is a bar chart chart type; 0.25 is used to characterize the probability that the training image is a pie chart chart type; 0.1 is used to characterize the probability that the training image is a table chart type.

[0051] For some implementations, a multi-layer perceptron (MLP) may be used as an auxiliary head to generate an auxiliary classification result, which is a first classification feature vector obtained by classifying each initial feature vector. For example, the auxiliary classification result may be represented by aux_output.

[0052] Optionally, the first classification feature vector can be represented by one-hot encoding. The advantage of one-hot encoding is that each category becomes independent, so that the model does not introduce any sequential information when processing these categories. That is, the use of one-hot encoding in the embodiment of the present application can reduce the mutual interference between different image types and improve the accuracy of image type recognition to a certain extent.

[0053] Step S130: performing a perturbation operation on each of the initial feature vectors to obtain a perturbed vector corresponding to each of the initial feature vectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial feature vector.

[0054] From the foregoing introduction, it can be seen that in order to improve the robustness of the initial model, a perturbation operation can be introduced in the training process of the initial model. By performing a perturbation operation on each of the initial feature vectors, a perturbed vector corresponding to each of the initial feature vectors is obtained, and the standard chart type corresponding to the perturbed vector should be the same as the standard chart type corresponding to the training image corresponding to the perturbed vector. In the subsequent training process, the initial model can more accurately learn the features of the training image and the perturbed vector, reducing the probability of incorrectly recognizing the perturbed vector in the subsequent use of the trained initial model, thereby improving the robustness.

[0055] Exemplarily, if a standard chart type corresponding to a training image is a bar chart, then a perturbation vector obtained by subsequently perturbing an initial feature vector corresponding to the training image should also correspond to a bar chart.

[0056] The perturbation operation may be performed on each initial feature vector, for example, the initial feature vector may be perturbed by a perturbation vector to obtain a perturbed vector. The perturbed vector corresponding to each initial feature vector is different from the initial feature vector. For the relevant introduction on obtaining the perturbation vector, please refer to the subsequent embodiments.

[0057] Step S140: Obtain a second classification feature vector for each of the disturbed vectors.

[0058] Further, after obtaining the disturbance vector, the second classification feature vector of each disturbance vector can be obtained. Similar to the first classification feature vector, the second classification feature vector of each disturbance vector can be obtained through the initial model. The second classification feature vector can be used to characterize the probability of each chart type to which the corresponding disturbance vector belongs.

[0059] For some implementations, the second classification feature vector can also be obtained by classification by the multi-layer perceptron in the aforementioned step. The multi-layer perceptron of the initial model can be used to perform a chart type recognition operation on each disturbed vector, thereby determining the second classification feature vector of each of the disturbed vectors.

[0060] Step S150: Based on the first difference between the first classification feature vector and the corresponding standard chart type, and the second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, the initial model is trained so that the first difference and the second difference obtained by the initial model converge to a target range.

[0061] It can be understood that when the chart type recognition operation is performed on the initial feature vector, a first classification feature vector is obtained, and each first classification feature vector has a first difference with the standard chart type corresponding to the initial feature vector. The smaller the first difference is, the higher the accuracy of the first classification feature vector obtained by performing the chart type recognition operation on the initial feature vector through the initial model. Therefore, the initial model can be trained by the first difference to reduce the first difference.

[0062] In addition, the second classification feature vector obtained by performing a chart type recognition operation on the disturbed vector has a second difference between each second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector. Exemplarily, if the standard chart type corresponding to the training image is a bar chart, then the disturbance vector obtained after the initial feature vector corresponding to the training image is subsequently perturbed, the standard chart type corresponding to the disturbance vector should also be a bar chart. Thus, the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector of the disturbance vector is a bar chart. The difference between the second classification feature vector and the bar chart is the second difference corresponding to the second classification feature vector. The smaller the second difference, the higher the accuracy of the second classification feature vector obtained by performing a chart type recognition operation on the disturbed vector through the initial model. Thus, the initial model can be trained by the second difference to reduce the second difference.

[0063] For some embodiments, the initial model may be trained based on the first difference and the second difference, so that the first difference and the second difference obtained by the trained initial model converge to a target range. The target range may include target ranges corresponding to each difference, for example, the first difference may correspond to a first target range, and the second difference may correspond to a second target range. Thus, if the first difference converges to the first target range, the second difference converges to the second target range, and the training of the initial model can be stopped.

[0064] Optionally, it can be preset that the training of the initial model is considered completed when the training batch reaches a specified number of times. The specified number of times can be determined empirically, so that after the initial model is trained for a specified number of batches, the first difference can generally converge to the first target range, and the second difference can generally converge to the first target range.

[0065] The model training method provided in the embodiment of the present application first extracts features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image; then obtains a first classification feature vector for each of the initial feature vectors; performs a perturbation operation on each of the initial feature vectors to obtain a perturbed vector corresponding to each of the initial feature vectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial feature vector; then obtains a second classification feature vector for each of the perturbed vectors; thereby training the initial model based on the first difference between the first classification feature vector and the corresponding standard chart type, and the second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to the target range. In the process of identifying the chart type of an image, the image may be subject to some disturbances, so that the perturbed image may be mistakenly identified as other chart types, that is, the chart type is incorrectly identified, and the robustness is low. In the present application, by performing a perturbation operation on the initial feature vector and then training the initial model based on the first difference and the second difference, the initial model can overcome the impact of the perturbation operation and more accurately learn the features in each training image, thereby improving the robustness of the trained initial model during use and the accuracy of chart recognition.

[0066] See also Figure 3 , Figure 3 A method flow chart of a model training method provided in an embodiment of the present application is shown. The model training method can be applied to Figure 1 The electronic device in the video generation scene shown can specifically use the processor of the electronic device as the execution subject of the model training method. The model training method can include steps S210 to S270.

[0067] Step S210: performing feature extraction on each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image.

[0068] Step S220: Obtain a first classification feature vector for each of the initial feature vectors.

[0069] Among them, step S210 and step S220 have been introduced in detail in the above embodiments and will not be repeated here.

[0070] Step S230: constructing a disturbance vector corresponding to the initial feature vector based on each of the first classification feature vectors.

[0071] Step S240: perturb each initial eigenvector using the perturbation vector corresponding to the initial eigenvector to obtain a perturbed vector corresponding to each initial eigenvector.

[0072] For some embodiments, a perturbation vector can be constructed by performing a chart type recognition operation on the initial feature vector to obtain a first classification feature vector, and then the initial feature vector is perturbed by the perturbation vector. It can be understood that the constructed perturbation vector corresponds to each initial feature vector, that is, the initial feature vector is subsequently perturbed by the perturbation vector corresponding to the initial feature vector. Exemplarily, for the initial feature vector A and the initial feature vector B, the first classification feature vector of the initial feature vector A is obtained as A1, and the first classification feature vector of the initial feature vector A is B1. Thus, a perturbation vector A2 corresponding to the first classification feature vector A1 and a perturbation vector B2 corresponding to the first classification feature vector B1 can be obtained. Therefore, the first feature vector A1 can be perturbed by the perturbation vector A2, and the first feature vector B1 can be perturbated by the perturbation vector B2.

[0073] As can be seen from the above introduction, the training image corresponds to a standard chart type, and the initial feature vector of each training image also corresponds to the standard chart type. Therefore, the perturbation vector can be constructed by the first classification feature vector of each initial feature vector and the standard chart type corresponding to the initial feature vector. Specifically, step S230 can also include step S231 and step S232.

[0074] Step S231: constructing a first loss function of each initial feature vector through the first classification feature vector of the initial feature vector and the standard chart type corresponding to the initial feature vector, wherein the first loss value of the initial feature vector determined based on the first loss function is positively correlated with the first difference corresponding to the first classification feature vector of the initial feature vector.

[0075] Step S232: construct a disturbance vector based on the first loss function.

[0076] First, the first loss function of the initial feature vector can be constructed by the first classification feature vector of each initial feature vector and the standard chart type corresponding to the initial feature vector, and then the perturbation vector can be constructed based on the first loss function. Among them, the first loss value of the initial feature vector determined based on the first loss function is positively correlated with the first difference corresponding to the first classification feature vector of the initial feature vector. Exemplarily, taking the initial feature vector as A and the first classification feature vector of the initial feature vector A as A1 as an example, the first loss value of the initial feature vector A determined based on the first loss function is A3, and the first difference corresponding to the first classification feature vector A1 is A4, then the first loss value A3 is positively correlated with the first difference A4. In other words, the larger the first difference corresponding to the initial feature vector, the larger the first loss value of the initial feature vector determined by the first loss function. Among them, the first difference corresponding to the initial feature vector is the first difference corresponding to the first loss function of the initial feature vector.

[0077] Therefore, after the initial model is trained, the first classification feature vector of the training image obtained will change, and the constructed perturbation vector will also change, so that in different training batches, even for the same training image, the perturbed vector constructed is different, so that the initial model can learn more features of the perturbed vector during the training process, improve the resistance of the trained initial model to the perturbation of the image to be recognized in the subsequent use process, force the initial model to learn a more robust feature representation, and make the trained initial model more difficult to be deceived by adversarial examples, thereby improving the generalization of the robustness of the initial model. Among them, adversarial examples refer to the image to be recognized formed by deliberately adding perturbations to the input of the initial model. Such perturbations are generally difficult for humans to detect, but can cause the model to give erroneous outputs with a high degree of confidence.

[0078] For some embodiments, the first loss function may be a cross-entropy loss function, so that the perturbation vector may be constructed based on the gradient of the cross-entropy loss function. Specifically, the gradient of the cross-entropy loss function may be first obtained; then the quotient of the gradient and the norm of the gradient may be obtained to obtain a first intermediate number; and then the perturbation vector may be obtained based on the product of a pre-set hyperparameter and the first intermediate number.

[0079] The disturbance vector can be represented by the following formula (1).

[0080]

[0081] Among them, disturbance is used to characterize the disturbance vector; g is used to characterize the gradient of the disturbance vector; ||g|| is used to characterize the norm of the gradient; ε is used to characterize the hyperparameter. Among them, the hyperparameter can be used to control the size of the disturbance, which can generally be set manually in advance. Among them, the norm of the gradient is a measure of the size of the gradient, and the norm is used to quantify the strength or size of the gradient.

[0082] Thus, after obtaining the perturbation vector, the initial feature vector can be perturbed by the perturbation vector corresponding to each initial feature vector to obtain the perturbed vector corresponding to each initial feature vector. Thus, the robustness of the initial model is improved by the perturbation vector determined based on the gradient. For some embodiments, step S240 may further include step S241.

[0083] Step S241: summing the disturbance vector corresponding to each initial eigenvector with the initial eigenvector to obtain the disturbed vector corresponding to each initial eigenvector.

[0084] That is, the disturbance vector corresponding to each initial eigenvector may be summed with the initial eigenvector to add disturbance to the initial eigenvector and obtain the disturbed vector corresponding to each initial eigenvector.

[0085] Optionally, each initial eigenvector may be subtracted from a disturbance vector corresponding to the initial eigenvector to obtain a disturbed vector.

[0086] Step S250: Obtain a second classification feature vector for each of the disturbed vectors.

[0087] Among them, step S250 has been introduced in detail in the above embodiment and will not be repeated here.

[0088] Step S260: constructing a second loss function based on each of the disturbed vectors, the initial feature vector, and the standard chart type corresponding to the disturbed vector, wherein a second loss value of the disturbed vector determined based on the second loss function is positively correlated with the second difference corresponding to the second classification feature vector of the disturbed vector.

[0089] Step S270: Based on the first loss function and the second loss function, the initial model is trained so that the first loss value of the initial feature vector and the second loss value of the perturbed vector obtained by the trained initial model converge to the target range corresponding to each loss value.

[0090] Furthermore, a second loss function can be constructed, and then the initial model can be trained based on the first loss function and the second loss function. The second loss value of the disturbed vector determined by the second loss function is positively correlated with the second difference corresponding to the second classification feature vector of the disturbed vector. Exemplarily, taking the disturbed vector C as an example, the second loss value of the disturbed vector C is C1, and the second difference corresponding to the second classification feature vector C1 is C2, then the second loss value C1 is positively correlated with the second difference C2. In other words, the larger the second difference corresponding to the disturbed vector, the larger the second loss value of the disturbed vector determined by the second loss function.

[0091] For some implementations, a second loss function may be constructed based on each of the perturbed vectors, the initial feature vectors, and the standard graph type corresponding to the perturbed vector. Exemplarily, the second loss function may be represented by the following formula (2).

[0092]

[0093] Among them, L InfoNCE is used to represent the second loss function, q is used to represent the initial feature vector of a training image; and k + Used to characterize the perturbed vector corresponding to the initial eigenvector, k i It is used to characterize the perturbed vector corresponding to each initial eigenvector. + There are the same standard chart types for the corresponding ones.

[0094] In some implementations, the second loss function may also be referred to as an Info NCE loss function.

[0095] Therefore, by guiding the initial model to be trained through the second loss function, the initial model can better learn that the initial feature vector and the perturbed vector corresponding to the initial feature vector belong to the same standard graph type.

[0096] Furthermore, based on the first loss function and the second loss function, the initial model is trained so that the first loss value of the initial feature vector and the second loss value of the perturbed vector obtained by the trained initial model converge to the target range corresponding to each loss value.

[0097] As can be seen from the above introduction, the target range can also include target ranges corresponding to each loss value. Thus, the initial model is trained to reduce the first loss value of the initial feature vector and the second loss value of the perturbed vector, so that the first loss value of the initial feature vector converges within the target range corresponding to the first loss value, and the second loss value of the perturbed vector converges within the target range corresponding to the second loss value.

[0098] Optionally, a third loss function may be further constructed, and then combined with the third loss function to guide the training of the initial model. Specifically, step S270 may also include steps S271 to S274.

[0099] Step S271: respectively obtaining a standard chart type different from the standard chart type corresponding to each first classification feature vector as a candidate standard chart type corresponding to each first classification feature vector.

[0100] Step S272: Determine the similarity between the standard chart type corresponding to each first classification feature vector and each candidate standard chart type as the target similarity corresponding to the first classification feature vector.

[0101] Step S273: constructing a third loss function based on the target similarity, wherein a third loss value of the first classification feature vector determined based on the third loss function is positively correlated with each of the target similarities corresponding to the first classification feature vector.

[0102] Step S274: Train the initial model based on the first loss function, the second loss function and the third loss function, so that the first loss value of the initial feature vector, the second loss value of the perturbed vector and the third loss value of the first classification feature vector obtained by the trained initial model all converge within the target range corresponding to each loss value.

[0103] For some embodiments, by performing a chart type recognition operation on the training images through the initial model, the first classification feature vectors of multiple training images are obtained, so that each first classification feature vector belonging to the same standard chart type can be close to each other, while the first classification feature vectors belonging to different standard chart types can be kept away from each other. Exemplarily, if the standard chart types include mind maps, bar charts, pie charts, and tables, the first classification feature vectors of the training images corresponding to the mind maps are expected to be close to each other; similarly, the first classification feature vectors of the training images corresponding to the bar charts are expected to be close to each other; the first classification feature vectors of the training images corresponding to the pie charts are expected to be close to each other; and the first classification feature vectors of the training images corresponding to the tables are expected to be close to each other. Furthermore, the first classification feature vectors of the training images corresponding to the mind maps, bar charts, pie charts, and tables, respectively, are expected to be far away from each other.

[0104] Thus, first, standard chart types different from the standard chart types corresponding to each first classification feature vector can be obtained respectively as the alternative standard chart types corresponding to each first classification feature vector. Exemplarily, the quasi-chart types including mind maps, bar charts, pie charts and tables are continued to be explained as an example. If the mind map corresponds to the first classification feature vector A1, the bar chart corresponds to the first classification feature vector B1, the pie chart corresponds to the first classification feature vector D1, and the chart corresponds to the first classification feature vector E1. Thus, the alternative standard chart types corresponding to the first classification feature vector A1 include bar charts, pie charts and tables; similarly, the alternative standard chart types corresponding to the first classification feature vector B1 include mind maps, pie charts and tables; the alternative standard chart types corresponding to the first classification feature vector D1 include mind maps, bar charts and tables; the alternative standard chart types corresponding to the first classification feature vector E1 include mind maps, bar charts and pie charts.

[0105] Further, the similarity between the standard chart type corresponding to each first classification feature vector and each candidate standard chart type is determined as the target similarity corresponding to the first classification feature vector. Exemplarily, the candidate standard chart types corresponding to the first classification feature vector A1 include a bar chart, a pie chart, and a table. The similarity between the first feature vector A1 and the bar chart, the pie chart, and the table is used as the target similarity corresponding to the first feature vector A1.

[0106] Thus, a third loss function can be constructed based on the target similarity, wherein the third loss value of the first classification feature vector determined based on the third loss function is positively correlated with each of the target similarities corresponding to the first classification feature vector. Exemplarily, if the third loss value of the first classification feature vector A1 determined based on the third loss function is A5, each of the target similarities corresponding to the first classification feature vector includes the similarity S1 between the first feature vector A1 and the bar chart, the similarity S2 between the first feature vector A1 and the pie chart, and the similarity S3 between the first feature vector A1 and the table, so that the third loss value is positively correlated with the similarity S1, the similarity S2, and the similarity S3, respectively. In other words, the greater the similarity S1, the similarity S2, and the similarity S3, the greater the third loss value A5 of the first classification feature vector A1 determined by the third loss function.

[0107] For some implementations, the third loss function may be characterized by equation (3).

[0108]

[0109] Among them, L SupCon is used to represent the third loss function, s(·) is used to represent the similarity operation, where y is used to represent the type, and the specific y iis a standard chart type used to iterate over all sample images, y j is the standard chart type of sample images traversed except the standard chart type of the i-th sample image, k is the standard chart type of sample images traversed except the standard chart type of the i-th sample image; z is used to characterize the first classification feature vector, which is similar to q and k in the above formula (2).

[0110] For some implementations, the third loss function may also be referred to as a SupCon loss function.

[0111] Through the third loss function, the standard chart types can be better utilized to train the initial model, so that the first classification feature vectors belonging to the same standard chart type are close to each other, and the first classification feature vectors belonging to different standard chart types are far away from each other.

[0112] Thus, the initial model can be trained based on the first loss function, the second loss function and the third loss function, so that the first loss value of the initial feature vector, the second loss value of the perturbed vector and the third loss value of the first classification feature vector obtained by the trained initial model all converge within the target range corresponding to each loss value.

[0113] From the above introduction, it can be known that the target range can also include target ranges corresponding to each loss value. Thus, the initial model is trained to reduce the first loss value of the initial feature vector, the second loss value of the perturbed vector, and the third loss value of the first classification feature vector, so that the first loss value of the initial feature vector converges to the target range corresponding to the first loss value, the second loss value of the perturbed vector converges to the target range corresponding to the second loss value, and the third loss value of the first classification feature vector converges to the target range corresponding to the third loss value. Thus, the initial model is jointly optimized by using cross entropy loss, InfoNCE loss, and SupCon loss.

[0114] The model training method provided in the embodiment of the present application constructs a perturbation vector through the gradient of the cross entropy loss function, and perturbs the first classification feature vector through the perturbation vector to obtain the perturbed vector, so that the initial model can learn more features of the perturbed vector during the training process, improve the resistance of the trained initial model to the perturbation of the image to be recognized in the subsequent use process, force the initial model to learn a more robust feature representation, and make the trained initial model more difficult to be deceived by adversarial samples, thereby improving the generalization of the robustness of the initial model. In addition, by guiding the initial model to train through the second loss function, the initial model can better learn that the initial feature vector and the perturbed vector corresponding to the initial feature vector belong to the same standard chart type. Furthermore, through the third loss function, the standard chart type can be better used to train the initial model, so that each first classification feature vector belonging to the same standard chart type is close to each other, and the first classification feature vectors belonging to different standard chart types are far away from each other, thereby realizing the joint optimization of the initial model by using cross entropy loss, InfoNCE loss and SupCon loss.

[0115] See also Figure 4 , Figure 4 A method flow chart of a model training method provided in an embodiment of the present application is shown. The model training method can be applied to Figure 1 The electronic device in the video generation scene shown can specifically use the processor of the electronic device as the execution subject of the model training method. The model training method can include steps S310 to S390.

[0116] Step S310: randomly selecting two initial images from the initial images of the training set multiple times to obtain multiple groups of image sets, wherein each group of image sets is different, and one initial image in each group of image sets is used as a source image, and the other initial image is used as a target image.

[0117] Step S320: reducing the size of the source image in each image set, and covering the reduced source image to a designated area of ​​a target image that belongs to the same image set as the source image, to obtain an extended image.

[0118] Step S330: taking each of the extended images and the initial image as training images.

[0119] In some implementations, the training set may include multiple initial images, wherein the method for obtaining the initial images can refer to the introduction in the aforementioned embodiment and will not be described in detail here.

[0120] Furthermore, the initial image can be processed by resizemix to further expand the training images in the training set. First, two initial images can be randomly selected from the initial images in the training set multiple times to obtain multiple sets of image sets. Each set of image sets is different, and one initial image in each set of image sets is used as the source image and the other initial image is used as the target image.

[0121] Exemplarily, if the training set includes initial image i1, initial image i2 and initial image i3, then image set 1 including initial image i1 and initial image i2; image set 2 including initial image i1 and initial image i3; and image set 3 including initial image i2 and initial image i3 can be obtained.

[0122] Among them, the initial image i1 in the above-mentioned image set 1 can be used as a source image, and the initial image i2 in the image set 1 can be used as a target image; the initial image i1 in the image set 2 can be used as a source image, and the initial image i3 in the image set 2 can be used as a target image; the initial image i2 in the image set 3 can be used as a source image, and the initial image i3 in the image set 3 can be used as a target image. It should be noted that the above-mentioned method of selecting source images and target images is only an example, and other selection methods can also be used in actual applications, and the embodiments of the present application do not specifically limit it.

[0123] Furthermore, the size of the source image in each group of image sets can be reduced, and the reduced source image can be overlaid on a specified area of ​​a target image that belongs to the same image set as the source image to obtain an extended image. The following description will be continued by taking the initial image i1 in image set 1 as the source image and the initial image i2 in image set 1 as the target image. Specifically, the size of the initial image i1 can be changed, such as by reducing it, and then the reduced source image i1 can be overlaid on the target image i2, wherein the overlaid target image is a target image that belongs to the same image set as the reduced source image. Thus, the overlaid target image i2 can be obtained as an extended image, thereby achieving the expansion of the image data.

[0124] Therefore, by expanding the image data of each set of images in the above manner, an expanded image corresponding to each set of images can be obtained. Thus, each of the expanded images and the initial image can be used as a training image. In other words, the training images in the training set include the acquired initial images and the expanded images generated from the initial images, thereby achieving the expansion of the training images in the training set.

[0125] Optionally, the extended image obtained above may be set to have no corresponding standard chart type, that is, the standard chart type corresponding to the extended image is unknown, thereby further improving the recognition capability of the chart type of the initial model image.

[0126] Optionally, after the initial images are acquired by the method described in the above embodiment, image transformation processing can be performed on each initial image to expand the training set to obtain more initial images, and then the expanded images are generated by more initial images. Exemplarily, step S310 can also include steps S311 to S313.

[0127] Step S311: performing image transformation processing on each of the initial images.

[0128] Step S312: adding the initial image after the image transformation processing as a new initial image to the training set to update the training set.

[0129] Step S313: randomly selecting two initial images from the initial images of the updated training set multiple times to obtain multiple sets of images.

[0130] For some embodiments, each initial image is first subjected to image transformation processing. The image transformation processing includes a transformation operation performed on the initial image and the strength of the transformation operation. Exemplarily, a thumbnail operation may be performed on the initial image, and the strength of the thumbnail operation is relatively strong.

[0131] The image transformation process may include a variety of transformation operations, such as image reduction operation, image expansion operation, rotation, cropping, color transformation, blurring, etc. It is understandable that the image transformation process performed on each initial image may be to randomly select a transformation operation and determine the intensity corresponding to the transformation operation.

[0132] In some embodiments, each initial image may be subjected to image transformation processing by using the RandAug random data enhancement strategy. In the process of executing the RandAug random data enhancement strategy, an operation pool may be predefined, and the operation pool includes a plurality of transformation operations, so that for each initial image, a transformation operation may be selected from the operation pool, and the selected transformation operation may be applied to the initial image with a random intensity to obtain the initial image after image transformation processing.

[0133] Optionally, the RandAug random data enhancement strategy may be used to perform image transformation processing on each initial image multiple times, that is, multiple initial images after image transformation processing may be obtained for one initial image. For example, the image transformation processing may be performed twice.

[0134] Then, the initial image after the image transformation process may be added to the training set as a new initial image to update the training set, thereby expanding the training set.

[0135] Thus, two initial images can be selected arbitrarily multiple times from the initial images of the updated training set to obtain multiple sets of image sets. The method of determining the image set is similar to the above, and will not be repeated here.

[0136] Optionally, the image can be converted into a corresponding tensor representation through the RandAug random data enhancement strategy, that is, the tensor representation of each initial image and the transformed initial image can be obtained. Then, when two initial images are randomly selected from the initial images of the updated training set multiple times to obtain multiple sets of images, the tensor representation of each initial image and the transformed initial image can be directly processed to obtain the tensor representation of the extended image, and then the feature extraction network is used to extract features from each tensor representation to obtain the first feature vector.

[0137] Step S340: extracting features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image.

[0138] Step S350: Obtain a first classification feature vector of each of the initial feature vectors.

[0139] Step S360: performing a perturbation operation on each of the initial eigenvectors to obtain a perturbed vector corresponding to each of the initial eigenvectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial eigenvector.

[0140] Step S370: Obtain a second classification feature vector for each of the disturbed vectors.

[0141] Among them, steps S340 to S370 have been introduced in detail in the above embodiments and will not be repeated here.

[0142] Step S380: Based on the first difference and the second difference, the initial model is trained. When the first difference and the second difference obtained by the trained initial model converge to a target range, the trained initial model is used as a model to be verified.

[0143] Step S390: fine-tune the model to be verified to obtain the chart recognition model, wherein the chart recognition model is used to determine the classification probability that the image to be recognized belongs to each chart type, and when the highest classification probability is greater than the probability threshold corresponding to the highest classification probability, determine that the chart type of the image to be recognized is the chart type corresponding to the highest classification probability.

[0144] For some implementations, the initial model is trained based on the first difference and the second difference, so that when the first difference and the second difference obtained by the trained initial model converge to the target range, the trained initial model is used as the model to be verified. The introduction of training the initial model based on the first difference and the second difference can be referred to in the above embodiment, which will not be repeated here.

[0145] After obtaining the model to be verified, the model to be verified can be further fine-tuned to obtain a graph recognition model. For example, the relevant parameters of the feature extraction network in the model to be verified can be frozen, and then a fully connected layer with SpectralNorm is added to the model to be verified as a classification head, and then the graph classification task is trained using the training set obtained above, and the trained model to be verified is used as a graph recognition model. For example, please refer to Figure 5 , Figure 5 A flow chart of a method for fine-tuning a model to be verified provided in an embodiment of the present application is shown. Figure 5 Specifically includes steps S391 to S394.

[0146] Step S391: Obtain a training set.

[0147] Step S392: Feature extraction network.

[0148] Step S393: Feature embedding representation.

[0149] Step S394: Classification header.

[0150] After the sample images in the training set are input into the feature extraction network, the input of the feature extraction network is used as input and input into the feature embedding representation to obtain the initial feature vector of each sample image. The chart type is then classified through the classification head, and the trained model to be verified is used as the chart recognition model.

[0151] As another exemplary method, the weights of each network layer in the model to be verified may be directly loaded, and then the weights and other parameters of the auxiliary head may be updated to obtain a graph recognition model.

[0152] The chart recognition model is used to determine the classification probability that the image to be recognized belongs to each chart type, and when the highest classification probability is greater than the probability threshold corresponding to the highest classification probability, the chart type of the image to be recognized is determined to be the chart type corresponding to the highest classification probability.

[0153] Exemplarily, in the subsequent process of using the chart recognition model, if the image to be recognized is used as input, the classification probability of the image to be recognized belonging to each chart type can be obtained through the chart recognition model. Specifically, for example, the classification probability of the image to be recognized belonging to each chart type is represented by the vector [0.3, 0.2, 0.25, 0.1], where 0.3 is used to represent the probability that the training image is a chart type of a mind map; 0.2 is used to represent the probability that the training image is a chart type of a bar chart; 0.25 is used to represent the probability that the training image is a chart type of a pie chart; 0.1 is used to represent the probability that the training image is a chart type of a table. Among them, 0.3 is the highest classification probability, then the probability threshold corresponding to the mind map can be obtained at this time. If the probability threshold is 0.1, then it is determined that the highest classification probability is greater than the probability threshold corresponding to the highest classification probability, so that the chart type of the image to be recognized can be determined to be the chart type corresponding to the highest classification probability, that is, the chart type of the image to be recognized is a mind map. If the probability threshold is 0.5, then it is determined that the highest classification probability is not greater than the probability threshold corresponding to the highest classification probability, so that the chart type of the image to be identified can be determined to be a non-chart type image. In other words, the chart recognition model obtained after training does not only perform closed set recognition, but can also perform open set recognition, so that images that do not belong to the standard chart type can be identified as non-chart type images.

[0154] Optionally, the probability threshold corresponding to each classification probability, that is, the probability threshold of the chart type corresponding to each classification probability, can be statistically determined based on the data in the process of testing the chart recognition model. Specifically, the chart recognition model can be tested through a test set, which can include multiple test images and a standard chart type corresponding to each test image. The chart recognition model performs a chart type recognition operation on each test image, and determines the accuracy of the chart recognition model for each chart type as the probability threshold corresponding to the chart type.

[0155] Among them, for a standard chart type, a first number of standard chart types correctly identified by the chart recognition model can be obtained, and then a second number of test images belonging to the chart type in the test set can be obtained, and the ratio of the first number to the second number is used as the probability threshold corresponding to the standard chart type.

[0156] The model training method provided in the embodiment of the present application expands the image data of each group of image sets in the above-mentioned manner, and an extended image corresponding to each group of image sets can be obtained. Therefore, each of the extended images and the initial image can be used as a training image. In other words, the training images in the training set include the acquired initial images and the extended images generated by each initial image, thereby achieving the expansion of the training images in the training set. In addition, the initial image after image transformation processing can also be added to the training set as a new initial image to update the training set to achieve the expansion of the training set. By expanding the training set, the trained initial model can better identify the chart type of the image to be identified and has better generalization ability.

[0157] See also Figure 6 , Figure 6 A method flow chart of a model training method provided in an embodiment of the present application is shown. The model training method can be applied to Figure 1 The electronic device in the video generation scene shown can specifically use the processor of the electronic device as the execution subject of the model training method. The model training method can include steps S410 to S490.

[0158] Step S410: Acquire at least one sample image that does not correspond to a standard chart type.

[0159] Step S420: respectively determining the cosine similarity between each of the sample images and each of the initial images.

[0160] Step S430: sorting the cosine similarities from high to low, and selecting a specified number of cosine similarities based on the sorting as target similarities.

[0161] Step S440: adding the sample image corresponding to the target similarity as a new initial image to the training set to update the training set.

[0162] It is understandable that if the training set only includes training images corresponding to standard chart types, the training of the initial model may be too easy, resulting in a weak recognition ability of the chart type of the image to be recognized by the trained initial model. Therefore, for some embodiments, negative samples can be added to the training set. The negative samples can represent sample images that do not correspond to standard chart types.

[0163] It should be noted that the sample image that does not correspond to a standard chart type means that the sample image is an image of a non-standard chart type, for example, the sample image does not include any chart.

[0164] Exemplarily, at least one sample image that does not correspond to a standard chart type may be first obtained. The method for obtaining the sample image that does not correspond to a standard chart type is similar to the method for obtaining the initial image described in the above embodiment, and will not be repeated here.

[0165] Then, the cosine similarity between each sample image and each initial image is determined respectively. The cosine similarity is used to characterize the similarity between the sample image and the initial image. The higher the cosine similarity, the higher the corresponding similarity.

[0166] It is understandable that the higher the similarity between the sample image and the initial image, the more difficult it is for the initial model to distinguish the sample image from the initial image. Therefore, the sample image with the highest similarity to the initial image can be found, and the sample image with the highest similarity can be added to the training set as a hard negative sample, so that the initial model can be trained by the training set including the hard negative sample, and the ability of the trained initial model to recognize the chart type of the image to be recognized can be further improved.

[0167] Specifically, the cosine similarities can be sorted from high to low, and a specified number of cosine similarities can be selected based on the sorting as the target similarity; then the sample images corresponding to the target similarity are added to the training set as new initial images to update the training set. The sample images corresponding to the target similarity are hard negative samples.

[0168] In some embodiments, the specified number may be a pre-set value, which may be a fixed value. In other embodiments, the specified number may also be determined based on the current batch for training the initial model. For example, the quotient of the value of the current batch and the number of standard chart types may be used as the specified value. Exemplarily, if the number of standard chart types is 4 and the current batch is 8, then 8 / 4=2, i.e., 2 is used as the specified value. Optionally, in the case where the specified value is a non-integer, the quotient of the value of the current batch and the number of standard chart types may be rounded up, and the result after the rounding may be used as the specified value. Exemplarily, if the number of standard chart types is 4 and the current batch is 10, then 10 / 4=2.5, and 3 may be used as the specified value at this time.

[0169] By introducing sample images to update the training set, the effect of the initial model comparison learning paradigm can be effectively improved to obtain better generalization performance, thus avoiding the initial model from fitting too easily and reducing the recognition ability of the chart type of the image to be recognized.

[0170] It is understandable that since training the initial model after updating the training set with hard negative samples may result in relatively high time complexity and computing power requirements, hard negative samples can also be selectively introduced according to the difficulty of the current task. For example, the hard negative samples can be introduced through a function to update the training set, and then a switch can be used to control whether the function is enabled, thereby achieving flexible control over whether the training set needs to be updated with hard negative samples.

[0171] Step S450: extracting features from each training image in the pre-acquired training set using the initial model to obtain an initial feature vector corresponding to each training image.

[0172] Step S460: Obtain a first classification feature vector of each of the initial feature vectors through the initial model.

[0173] Step S470: performing a perturbation operation on each of the initial eigenvectors to obtain a perturbed vector corresponding to each of the initial eigenvectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial eigenvector.

[0174] Step S480: Obtain a second classification feature vector of each disturbed vector through the initial model.

[0175] Step S490: Based on the first difference and the second difference, the initial model is trained so that the first difference and the second difference obtained by the trained initial model converge to a target range, wherein the first difference includes the difference between the first classification feature vector and the corresponding standard chart type, and the second difference includes the difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector.

[0176] Among them, steps S450 to S490 have been introduced in detail in the above embodiments and will not be repeated here.

[0177] The model training method provided in the embodiment of the present application can find the sample image with the highest similarity to the initial image, and the sample image with the highest similarity can be added to the training set as a difficult negative sample, so that the initial model is trained by the training set including the difficult negative sample, and the ability of the trained initial model to recognize the chart type of the image to be recognized can be further improved. In addition, the difficult negative sample can be introduced through a function to update the training set, and then the switch can be used to control whether the function is enabled, so as to achieve flexible control of whether the training set needs to be updated through difficult negative samples. In addition, the model training method provided in the embodiment of the present application does not require too much architecture design, and can be conveniently and effectively migrated to various application scenarios, such as scenarios for chart classification in offices.

[0178] See also Figure 7 , Figure 7 A method flow chart of a model training method provided in an embodiment of the present application is shown. The model training method can be applied to Figure 1 The electronic device in the video generation scene shown can specifically use the processor of the electronic device as the execution subject of the model training method. The model training method can include steps S510 to S5110.

[0179] Step S510: Obtain a training set.

[0180] The method for obtaining the training set can be found in the introduction of the aforementioned embodiment and will not be described in detail here.

[0181] Among them, the training set can be the initial image obtained by the method introduced in the aforementioned embodiment before executing the model training method provided in the embodiment of the present application.

[0182] Step S520: performing image transformation processing on each initial image in the training set.

[0183] The RandAug random data enhancement strategy can be used to perform image transformation processing on each initial image multiple times, that is, multiple initial images after image transformation processing can be obtained for one initial image. For example, the image transformation processing can be performed twice.

[0184] The initial image after the image transformation process can jump to step S530 and step S540 respectively.

[0185] For some implementations, the initial image after image transformation processing may be represented by images.

[0186] Step S530: deep copy operation.

[0187] Step S540: Mixing operation.

[0188] In step S530, a deep copy operation may be performed on the initial image after the image transformation process. A deep copy is a form of object copying, through which a new object may be created, and all properties of the original object may be recursively copied to the new object. Thus, in the embodiment provided by the present application, the original object includes the initial image after the image transformation process, and the new object includes an object having the same properties as the initial image after the image transformation process. Thus, the new object after the deep copy is completely independent of the original object, and modifying some of the objects will not affect other objects.

[0189] For some implementations, the image obtained after the deep copy operation can be represented by orig_images.

[0190] In step S540, a blending operation may be performed on the initial image after the image transformation process. The blending operation may be performed on the initial image after the image transformation process by resizemix. For some implementations, the blending operation is performed on the initial image after the image transformation process, and the resulting image may be represented by aug_images.

[0191] Step S550: extracting feature vectors through a feature extraction network.

[0192] Furthermore, feature extraction may be performed on the image after the deep copy operation in step S530 and the image after the blending operation in step S540 through a feature extraction network, thereby obtaining an initial feature vector corresponding to each image.

[0193] The feature extraction network can be a MobileNet V2 network, and a linear layer with SpectralNorm is used as the projection layer of the feature extraction network. Through the projection layer, the parameters of each layer in the neural network can be subjected to singular value decomposition (SVD), and the largest singular value is limited to 1, satisfying the 1-Lipschitz condition, that is, Lipschitz is 1. After each weight update, it is divided by its largest singular value. This can effectively limit the severity of the value change and make the gradient more stable when the weight is updated.

[0194] It can effectively limit the severity of numerical changes, making the gradient of weight updates more stable during subsequent training.

[0195] Step S560: auxiliary classification.

[0196] Step S570: initial feature vector of the initial image after image transformation processing.

[0197] Step S580: performing a perturbation operation on the initial feature vector of the initial image after the image transformation process to obtain a corresponding perturbed vector.

[0198] A multi-layer perceptron can be added to the initial model as an auxiliary head to generate an auxiliary classification result, which is the first classification feature vector obtained by classifying each initial feature vector. The auxiliary classification result can be represented by aux_output.

[0199] Then, a cross entropy loss function can be constructed through the first classification feature vector of each initial feature vector as the first loss function. A perturbation vector is then constructed based on the cross entropy loss function, and the perturbation vector is then fed back to the initial feature vector of the initial image after the image transformation processing in step S570 to perturb the initial feature vector of the initial image after the image transformation processing to obtain a perturbed vector. The initial feature vector of the initial image after the image transformation processing can be represented by Feature Embeddings Orig, and the perturbed vector can be represented by Feature Embeddings Mix.

[0200] Step S590: the initial image is subjected to image transformation processing and a blending operation to obtain an initial feature vector corresponding to the initial image.

[0201] Through the feature extraction network in step S550, the initial feature vector corresponding to the initial image after image transformation processing and mixing operation can also be obtained.

[0202] Step S5100: Calculate the second loss function.

[0203] Step S5110: Calculate the third loss function.

[0204] Thus, based on the perturbed vector obtained in step S580, the initial feature vector of the initial image after the image transformation processing obtained in step S570, and the initial feature vector corresponding to the image transformation processing and the mixing operation of the initial image obtained in step S590, a second loss function and a third loss function are constructed. Among them, the second loss function is the InfoNCE loss function, and the third loss function is the SupCon loss function. Thus, the initial model can be jointly trained with the cross entropy loss function, the InfoNCE loss function, and the SupCon loss function.

[0205] In the embodiment provided in the present application, since the object of the mixing operation is the initial image after the image transformation processing, only the initial feature vector of the initial image after the image transformation processing is perturbed by the perturbation vector, and there is no need to perform a perturbation operation on the image after the mixing operation, which can save computing power to a certain extent and improve the training efficiency of the model.

[0206] Please read Figure 8 , Figure 8 A flowchart of a diagram recognition method provided by an embodiment of the present application is shown. The diagram recognition method can be applied to Figure 1 The electronic device in the video generation scene shown can specifically use a processor of the electronic device as an execution subject for executing the chart recognition method. The chart recognition method can include step S610 and step S620.

[0207] Step S610: Acquire an image to be recognized.

[0208] Step S620: using a chart recognition model to recognize the image to be recognized, and outputting a chart type corresponding to the image to be recognized, wherein the chart recognition model is trained based on the model training method described in the aforementioned method embodiments.

[0209] For some implementations, the user can execute the chart recognition method through an electronic device, and the image to be recognized can be an image stored in the electronic device; it can also be an image obtained by the electronic device through an application, such as an image obtained through a browser search, or an image obtained through a chat application, etc., which is not specifically limited in the embodiments of the present application.

[0210] The user can obtain the image to be recognized through the electronic device, and then use the chart recognition model to recognize the image to be recognized, and output the chart type corresponding to the image to be recognized. For example, if the image to be recognized includes a mind map, the chart type of the image to be recognized can be obtained as a mind map type by recognizing the image to be recognized through the chart recognition model; if the image to be recognized does not include a chart, the chart type of the image to be recognized can be obtained as a non-chart type by recognizing the image to be recognized through the chart recognition model.

[0211] Among them, the above-mentioned chart recognition model is trained based on the model training method described in the aforementioned method embodiments. Specifically, the trained initial model in the embodiments corresponding to the aforementioned model training methods can be used as the chart recognition model, wherein the initial model is trained so that the first difference and the second difference obtained by the initial model converge to the target range. The detailed introduction of obtaining the chart recognition model can be found in the aforementioned method embodiments, which will not be repeated here.

[0212] See also Fig. 9 , Fig. 9 A structural block diagram of a model training device provided in an embodiment of the present application is shown. The model training device 900 includes: an extraction unit 910, a first feature vector acquisition unit 920, a perturbation unit 930, a second feature vector acquisition unit 940 and a training unit 950.

[0213] The extraction unit 910 is used to extract features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image.

[0214] Optionally, the extraction unit 910 can also be used to arbitrarily select two initial images from the initial images of the training set multiple times to obtain multiple groups of image sets, wherein each group of image sets is different, one initial image in each group of image sets is used as a source image, and the other initial image is used as a target image; reduce the size of the source image in each group of image sets, and cover the reduced source image to a specified area of ​​the target image belonging to the same image set as the source image to obtain an extended image; and use each of the extended images and the initial image as a training image.

[0215] Optionally, the extraction unit 910 can also be used to perform image transformation processing on each of the initial images; add the initial image after the image transformation processing as a new initial image to the training set to update the training set; and arbitrarily select two of the initial images from the initial images of the updated training set multiple times to obtain multiple groups of image sets.

[0216] Optionally, the extraction unit 910 can also be used to obtain at least one sample image that does not correspond to a standard chart type; determine the cosine similarity between each of the sample images and each of the initial images respectively; sort the cosine similarities from high to low, and based on the sorting, select a specified number of cosine similarities as target similarities; add the sample image corresponding to the target similarity as a new initial image to the training set to update the training set.

[0217] The first feature vector acquisition unit 920 is configured to acquire a first classification feature vector for each of the initial feature vectors.

[0218] The perturbation unit 930 is used to perform a perturbation operation on each of the initial eigenvectors to obtain a perturbed vector corresponding to each of the initial eigenvectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial eigenvector.

[0219] Optionally, the perturbation unit 930 can also be used to construct a perturbation vector corresponding to the initial feature vector based on each of the first classification feature vectors; perturb the initial feature vector using the perturbation vector corresponding to each initial feature vector to obtain a perturbed vector corresponding to each initial feature vector.

[0220] Optionally, the perturbation unit 930 can also be used to construct a first loss function of the initial feature vector through the first classification feature vector of each initial feature vector and the standard chart type corresponding to the initial feature vector, wherein the first loss value of the initial feature vector determined based on the first loss function is positively correlated with the first difference corresponding to the first classification feature vector of the initial feature vector; and construct a perturbation vector based on the first loss function.

[0221] Optionally, the perturbation unit 930 can also be used to obtain the gradient of the cross entropy loss function; obtain the quotient of the gradient and the norm of the gradient to obtain a first intermediate number; and obtain the perturbation vector based on the product of a pre-set hyperparameter and the first intermediate number.

[0222] Optionally, the perturbation unit 930 may also be configured to sum the perturbation vector corresponding to each initial eigenvector with the initial eigenvector to obtain a perturbed vector corresponding to each initial eigenvector.

[0223] The second feature vector acquisition unit 940 is used to acquire a second classification feature vector of each of the disturbed vectors through the initial model.

[0224] Optionally, the second feature vector acquisition unit 940 can also be used to train the initial model based on the first difference and the second difference, and when the first difference and the second difference obtained by the trained initial model converge to the target range, the trained initial model is used as the model to be verified; the model to be verified is fine-tuned to obtain the chart recognition model, wherein the chart recognition model is used to determine the classification probability that the image to be recognized belongs to each chart type, and when the highest classification probability is greater than the probability threshold corresponding to the highest classification probability, determine that the chart type of the image to be recognized is the chart type corresponding to the highest classification probability.

[0225] The training unit 950 is used to train the initial model based on the first difference between the first classification feature vector and the corresponding standard chart type, and the second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to a target range.

[0226] Optionally, the training unit 950 can also be used to construct a second loss function based on each of the disturbed vectors, the initial feature vectors and a standard chart type corresponding to the disturbed vector, wherein the second loss value of the disturbed vector determined based on the second loss function is positively correlated with the second difference corresponding to the second classification feature vector of the disturbed vector; based on the first loss function and the second loss function, the initial model is trained so that the first loss value of the initial feature vector and the second loss value of the disturbed vector obtained by the trained initial model converge to the target range corresponding to each loss value.

[0227] Optionally, the training unit 950 can also be used to obtain standard chart types different from the standard chart types corresponding to each first classification feature vector, as alternative standard chart types corresponding to each first classification feature vector; determine the similarity between the standard chart type corresponding to each first classification feature vector and each alternative standard chart type, as the target similarity corresponding to the first classification feature vector; construct a third loss function based on the target similarity, wherein the third loss value of the first classification feature vector determined based on the third loss function is positively correlated with each of the target similarities corresponding to the first classification feature vector; train the initial model based on the first loss function, the second loss function and the third loss function, so that the first loss value of the initial feature vector, the second loss value of the perturbed vector and the third loss value of the first classification feature vector obtained by the trained initial model all converge to the target range corresponding to each loss value.

[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0229] In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0230] See also Fig.10 , Fig.10 The block diagram of a structure of an electronic device provided in an embodiment of the present application is shown. The electronic device 110 may be a smart phone, a desktop computer, a vehicle-mounted computer, a server or a tablet computer, etc. The electronic device 110 in the present application may include one or more of the following components: a processor 111, a memory 112 and one or more application programs, wherein the processor 111 is electrically connected to the memory 112, and the one or more program configurations are used to execute the methods described in the aforementioned embodiments.

[0231] The processor 111 may include one or more processing cores. The processor 111 uses various interfaces and lines to connect various parts of the entire electronic device 110, and executes various functions and processes data of the electronic device 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 112, and calling data stored in the memory 112. Optionally, the processor 111 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 111 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and computer programs; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 111, and it can be implemented separately through a communication chip. Specifically, the method described in the above embodiment can be executed by one or more processors 111.

[0232] For some embodiments, the memory 112 may include a random access memory (RAM) or a read-only memory (ROM). The memory 112 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 112 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 110 during use, etc.

[0233] See also Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1100 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0234] The computer readable storage medium 1100 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1100 has storage space for program code 1110 that executes any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1110 can be compressed, for example, in an appropriate form.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: include: Perform feature extraction on each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image; Obtaining a first classification feature vector for each of the initial feature vectors; Performing a perturbation operation on each of the initial eigenvectors to obtain a perturbed vector corresponding to each of the initial eigenvectors, wherein the perturbation operation is used to make the perturbed vector different from the corresponding initial eigenvector; Obtaining a second classification feature vector for each of the disturbed vectors; Based on a first difference between the first classification feature vector and the corresponding standard chart type, and a second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, the initial model is trained so that the first difference and the second difference obtained by the initial model converge to a target range.

2. The method according to claim 1, characterized in that The performing a perturbation operation on each of the initial eigenvectors to obtain a perturbed vector corresponding to each of the initial eigenvectors includes: Constructing a disturbance vector corresponding to the initial feature vector based on each of the first classification feature vectors; The initial eigenvector is perturbed by the perturbation vector corresponding to each initial eigenvector to obtain the perturbed vector corresponding to each initial eigenvector.

3. The method according to claim 2, characterized in that The step of constructing a disturbance vector corresponding to the initial feature vector based on each of the first classification feature vectors includes: Constructing a first loss function of each initial feature vector by using a first classification feature vector of the initial feature vector and a standard chart type corresponding to the initial feature vector, wherein a first loss value of the initial feature vector determined based on the first loss function is positively correlated with a first difference corresponding to the first classification feature vector of the initial feature vector; A perturbation vector is constructed based on the first loss function.

4. The method according to claim 3, characterized in that The first loss function includes a cross entropy loss function, and constructing a perturbation vector based on the first loss function includes: Obtaining the gradient of the cross entropy loss function; Obtaining a quotient of the gradient and a norm of the gradient to obtain a first intermediate number; The disturbance vector is obtained based on the product of a preset hyperparameter and the first intermediate number.

5. The method according to claim 3, characterized in that: The training of the initial model based on the first difference between the first classification feature vector and the corresponding standard chart type, and the second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to a target range, includes: Constructing a second loss function based on each of the perturbed vectors, the initial feature vector, and a standard chart type corresponding to the perturbed vector, wherein a second loss value of the perturbed vector determined based on the second loss function is positively correlated with the second difference corresponding to the second classification feature vector of the perturbed vector; Based on the first loss function and the second loss function, the initial model is trained so that the first loss value of the initial feature vector and the second loss value of the perturbed vector obtained by the trained initial model converge to the target range corresponding to each loss value.

6. The method according to claim 5, characterized in that The initial model is trained based on the first loss function and the second loss function so that the first loss value of the initial feature vector and the second loss value of the perturbed vector obtained by the trained initial model converge to the target ranges corresponding to the loss values, including: Respectively obtaining a standard chart type different from the standard chart type corresponding to each first classification feature vector as an alternative standard chart type corresponding to each first classification feature vector; Determine the similarity between the standard chart type corresponding to each first classification feature vector and each candidate standard chart type as the target similarity corresponding to the first classification feature vector; Constructing a third loss function based on the target similarity, wherein a third loss value of the first classification feature vector determined based on the third loss function is positively correlated with each of the target similarities corresponding to the first classification feature vector; The initial model is trained based on the first loss function, the second loss function and the third loss function so that the first loss value of the initial feature vector, the second loss value of the perturbed vector and the third loss value of the first classification feature vector obtained by the trained initial model all converge within the target range corresponding to each loss value.

7. The method according to claim 2, characterized in that The perturbing the initial feature vector by using the perturbation vector corresponding to each initial feature vector to obtain the perturbed vector corresponding to each initial feature vector includes: The disturbance vector corresponding to each initial eigenvector is summed with the initial eigenvector to obtain the disturbed vector corresponding to each initial eigenvector.

8. The method according to claim 1, characterized in that The training set includes a plurality of initial images. Before extracting features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image, the method further includes: Randomly selecting two initial images from the initial images of the training set multiple times to obtain multiple sets of image sets, wherein each set of image sets is different, and one initial image in each set of image sets is used as a source image, and the other initial image is used as a target image; Reduce the size of the source image in each image set, and cover the reduced source image to a specified area of ​​a target image that belongs to the same image set as the source image, to obtain an extended image; Each of the extended images and the initial image is used as a training image.

9. The method according to claim 8, characterized in that The method comprises randomly selecting two initial images from the initial images of the training set for multiple times to obtain multiple sets of image sets, including: Performing image transformation processing on each of the initial images; Adding the initial image after the image transformation processing as a new initial image to the training set to update the training set; Two initial images are randomly selected multiple times from the initial images of the updated training set to obtain multiple sets of image sets.

10. The method according to claim 1, characterized in that The training set includes a plurality of initial images. Before extracting features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image, the method further includes: Obtaining at least one sample image that does not correspond to a standard chart type; respectively determining the cosine similarity between each of the sample images and each of the initial images; The cosine similarities are sorted from high to low, and based on the sorting, a specified number of cosine similarities are selected as target similarities; The sample image corresponding to the target similarity is added to the training set as a new initial image to update the training set.

11. The method according to claim 1, characterized in that: The training of the initial model based on the first difference between the first classification feature vector and the corresponding standard chart type, and the second difference between the second classification feature vector and the standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to a target range, includes: Based on the first difference and the second difference, the initial model is trained, and when the first difference and the second difference obtained by the trained initial model converge to a target range, the trained initial model is used as a model to be verified; The model to be verified is fine-tuned to obtain the chart recognition model, wherein the chart recognition model is used to determine the classification probability that the image to be recognized belongs to each chart type, and when the highest classification probability is greater than the probability threshold corresponding to the highest classification probability, determine that the chart type of the image to be recognized is the chart type corresponding to the highest classification probability.

12. A model training device, characterized in that: include: An extraction unit, used to extract features from each training image in the pre-acquired training set to obtain an initial feature vector corresponding to each training image; A first feature vector acquisition unit, used to acquire a first classification feature vector for each of the initial feature vectors; A perturbation unit, configured to perform a perturbation operation on each of the initial eigenvectors to obtain a perturbed vector corresponding to each of the initial eigenvectors, wherein the perturbation operation is configured to cause a difference between the perturbed vector and the corresponding initial eigenvector; A second feature vector acquisition unit, used to acquire a second classification feature vector of each disturbed vector; A training unit is used to train an initial model based on a first difference between the first classification feature vector and a corresponding standard chart type, and a second difference between the second classification feature vector and a standard chart type corresponding to the first classification feature vector corresponding to the second classification feature vector, so that the first difference and the second difference obtained by the initial model converge to a target range.

13. A diagram recognition method, characterized in that: include: Obtain an image to be recognized; The image to be identified is identified using a chart recognition model, and a chart type corresponding to the image to be identified is output, wherein the chart recognition model is trained based on the model training method according to any one of claims 1 to 11.

Images