Data processing method and scrap steel detection model construction method

Abstract

The embodiment of the present specification provides a data processing method and a scrap steel detection model construction method, wherein the data processing method comprises: obtaining a prediction result of an object training sample according to an initial neural network model, a differentiable initial data enhancement manner and an initial network structure; adjusting network parameters of the initial neural network model according to a loss function constituted by the prediction result and a labeled result; synchronously adjusting the initial data enhancement manner and the initial network structure of the initial neural network model according to an object verification sample and the network parameters; and in the case that the adjusted initial data enhancement manner and the initial network structure satisfy a preset ending condition, determining a target data enhancement manner and a target network structure. The method can obtain complementary data enhancement manners and network structures on corresponding data sets at one time, and the data enhancement manner and the network structure synchronously determined through the method are applied to the scrap steel detection model, which can greatly improve the accuracy of scrap steel detection.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a data processing method. BACKGROUND

[0002] In the industrial vision field, in order to obtain good defect detection effect, appropriate data augmentation needs to be performed on the training data due to the small amount of data that can be collected. In the application scenarios of the industrial vision field, the computer computing power used is usually not very high, and the size and computing power required by the detection model will greatly affect the competitiveness of the product. Therefore, searching for a good data augmentation method and an efficient detection model network structure is a common goal pursued.

[0003] The existing method is to use segmented search, that is, first fix the network structure to search for the data augmentation method; then fix the data augmentation method to search for the backbone structure of the detection model; then fix the data augmentation method and the backbone structure of the detection model to search for the neck structure of the detection model; then fix the data augmentation method, the backbone structure of the detection model, and the neck structure of the detection model to search for the head structure of the detection model; and finally combine the search results of the four parts to form a complete data augmentation method and network structure.

[0004] However, since the segmented search is used, it is difficult to obtain good search results, and the segmented search has a large amount of redundant training, which leads to a longer overall search time. SUMMARY

[0005] Therefore, the embodiments of the present specification provide a data processing method. One or more embodiments of the present specification also relate to a data processing apparatus, a scrap steel detection model construction method, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects in the prior art.

[0006] According to a first aspect of the embodiments of the present specification, a data processing method is provided, including:

[0007] determining object training samples, object validation samples, and an initial neural network model;

[0008] obtaining a prediction result of the object training samples according to the initial data augmentation method and the initial network structure in the initial neural network model that are differentiable;

[0009] adjusting network parameters of the initial neural network model according to a loss function constituted by the prediction result and a label result;

[0010] synchronously adjust the initial data augmentation manner and the initial network structure of the initial neural network model according to the object verification sample and the network parameter;

[0011] determine a target data augmentation manner and a target network structure in a case where the adjusted initial data augmentation manner and the adjusted initial network structure satisfy a preset ending condition.

[0012] According to a second aspect of an embodiment of the present specification, a data processing apparatus is provided, comprising:

[0013] a first determination module configured to determine an object training sample, an object verification sample, and an initial neural network model;

[0014] a result obtaining module configured to obtain a prediction result of the object training sample according to a differentiable initial data augmentation manner and an initial network structure in the initial neural network model;

[0015] a first adjustment module configured to adjust a network parameter of the initial neural network model according to a loss function constituted by the prediction result and a labeled result;

[0016] a second adjustment module configured to synchronously adjust the initial data augmentation manner and the initial network structure of the initial neural network model according to the object verification sample and the network parameter;

[0017] a second determination module configured to determine a target data augmentation manner and a target network structure in a case where the adjusted initial data augmentation manner and the adjusted initial network structure satisfy a preset ending condition.

[0018] According to a third aspect of an embodiment of the present specification, a scrap steel detection model construction method is provided, comprising:

[0019] determining a scrap steel image training sample, a scrap steel image verification sample, and an initial scrap steel detection model;

[0020] obtaining a prediction result of the scrap steel image training sample according to a differentiable initial data augmentation manner and an initial network structure in the initial scrap steel detection model;

[0021] adjusting a network parameter of the initial scrap steel detection model according to a loss function constituted by the prediction result and a labeled result;

[0022] synchronously adjusting the initial data augmentation manner and the initial network structure of the initial scrap steel detection model according to the scrap steel image verification sample and the network parameter;

[0023] In a case where the adjusted initial data augmentation manner and the initial network structure satisfy a preset ending condition, a target data augmentation manner and a target network structure are determined.

[0024] According to the target data augmentation manner and the target network structure, the scrap steel detection model is constructed.

[0025] According to a fourth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0026] a memory and a processor;

[0027] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method or the scrap steel detection model construction method.

[0028] According to a fifth aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method or the scrap steel detection model construction method.

[0029] According to a sixth aspect of an embodiment of the present specification, a computer program is provided, which, when executed in a computer, causes the computer to perform the steps of the data processing method or the scrap steel detection model construction method.

[0030] One embodiment of the present specification implements a data processing method and device, wherein the method comprises determining object training samples, object verification samples, and an initial neural network model; obtaining a prediction result of the object training samples according to a differentiable initial data augmentation manner and an initial network structure in the initial neural network model; adjusting network parameters of the initial neural network model according to a loss function constituted by the prediction result and a labeled result; synchronously adjusting the initial data augmentation manner and the initial network structure of the initial neural network model according to the object verification samples and the network parameters; and in a case where the adjusted initial data augmentation manner and the initial network structure satisfy a preset ending condition, determining a target data augmentation manner and a target network structure.

[0031] Specifically, the data processing method obtains complementary data augmentation manners and network structures on a corresponding data set at one time through a method of simultaneously searching data augmentation manners and network structures of a neural network model (such as a detection model, a classification model, a segmentation model, etc.), thereby obtaining a better search result, avoiding repeated training of network parameters multiple times, and greatly reducing search time. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1is a specific processing diagram of a data processing method applied to a target detection model provided by one embodiment of the present specification;

[0033] Figure 2 is a flowchart of a data processing method provided by one embodiment of the present specification;

[0034] Figure 3 is a search diagram of a data enhancement method in a data processing method provided by one embodiment of the present specification;

[0035] Figure 4 is a search diagram of a network backbone in a data processing method provided by one embodiment of the present specification;

[0036] Figure 5 is a whole search space diagram of a network neck in a data processing method provided by one embodiment of the present specification;

[0037] Figure 6 is a specific processing diagram of a network neck according to different OPs in a data processing method provided by one embodiment of the present specification;

[0038] Figure 7 is a processing diagram of feature fusion using TopDown in a network neck in a data processing method provided by one embodiment of the present specification;

[0039] Figure 8 is a whole search diagram of a network head in a data processing method provided by one embodiment of the present specification;

[0040] Figure 9 is a specific processing diagram of a network head according to different OPs in a data processing method provided by one embodiment of the present specification.

[0041] Figure 10 is a structural diagram of a data processing device provided by one embodiment of the present specification;

[0042] Figure 11 is a structural block diagram of a computing device provided by one embodiment of the present specification. DETAILED DESCRIPTION

[0043] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, set forth in this description. Those skilled in the art, in light of the description, can implement the present specification without limiting the scope of the present specification.

[0044] The terminology used in this disclosure one or more embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure one or more embodiments. As used in this disclosure one or more embodiments and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification one or more embodiments, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] It is to be understood that the singular forms "a," "an," and "the" used in this disclosure one or more embodiments are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification one or more embodiments, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0046] First, the noun terms related to the disclosure one or more embodiments are explained.

[0047] Bernoulli Distribution: Bernoulli Distribution, also known as two-point distribution or 0-1 distribution.

[0048] Categorical Distribution: Categorical Distribution is used to model the distribution of the maximum value in a set of sampled data.

[0049] backbone: translated as backbone network, its function is to extract information from the picture for the use of the network behind.

[0050] head: head is a network that obtains network output content, using the extracted features, head uses these features to make predictions.

[0051] neck: is placed between backbone and head, is to better use the features extracted by backbone.

[0052] Identity: direct connection, equal to no operation, input is equal to output.

[0053] FuseSplit: first fusion and then separation, that is, first fuse two layers into one layer, and then separate into two layers.

[0054] Three-dimensional video: three-dimensional video is to use three-dimensional software to establish a virtual world in the computer according to the actual demand, technical personnel in this virtual three-dimensional world according to the real size of the object to be displayed, build scene and establish model, make realistic material, light, configure camera motion effect, these are completed to let the computer operation production, post-editing output complete video.

[0055] Taking a detection model for completing a target detection task as an example, an existing search method searches the data enhancement mode of the detection model, the backbone of the detection model, the neck of the detection model and the head of the detection model respectively through a segmented search mode. However, the segmented search mode needs to repeatedly train network parameters and consumes more time. In addition, a good data enhancement mode is helpful to find a good network structure, and a good network structure is also helpful to find a better data enhancement mode.

[0056] Therefore, an embodiment of the specification provides a method for simultaneously searching a data enhancement mode and a network structure of a target detection model. A unified model is established for the data enhancement mode and the network structure, so that complementary data enhancement modes and network structures on a corresponding data set are obtained at one time.

[0057] In the specification, a data processing method is provided. The specification also relates to a data processing device, a scrap steel detection model construction method, a computing device and a computer readable storage medium, which are described in detail one by one in the following embodiments.

[0058] Referring to Figure 1 , Figure 1 A specific processing schematic diagram of a data processing method provided by an embodiment of the specification applied to a target detection model is shown.

[0059] As Figure 1 indicated, the architecture for simultaneously searching a data enhancement mode and a network structure in a target detection model includes a training set picture 102, a validation set picture 104, a data enhancement mode and a network structure of a target detection model 106: a first network (backbone), a second network (neck) and a third network (head).

[0060] In specific implementation, each search includes two steps. In the first step, the probability and amplitude of data enhancement and the model parameters theta of the network structure are fixed. K pictures (for example, K=32) are taken from the training set picture 102. A first loss function loss detto optimize the network parameter w, i.e. fix the network structure, optimize the network parameter; the second step is to fix the network parameter w, take K pictures (for example, K = 32) from the verification set pictures 104, and form a second loss function loss according to the prediction result and the label result of each verification set picture latency to optimize the probability, amplitude and network structure of data enhancement, i.e. fix the network parameter, optimize the probability, amplitude and network structure of data enhancement; the above two steps are repeatedly alternated until the network parameter w and the model parameter theta of the probability, amplitude and network structure of data enhancement are optimal, and the process is ended. At this time, the model parameter theta of the probability, amplitude and network structure of data enhancement is the final search result, i.e. the target data enhancement mode and the target network structure in the target detection model are searched synchronously, and the target detection model including the optimal model parameter theta of the probability, amplitude and network structure of data enhancement is the final target detection model.

[0061] Referring to Figure 2 , Figure 2 A flowchart of a data processing method according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0062] Step 202: Determine the object training sample, the object verification sample, and the initial neural network model.

[0063] In the embodiment of the present specification, the data enhancement mode and the network structure searched synchronously by the data processing method are different, and the initial neural network model used is also different according to different tasks to be processed; for example, in the case of a target detection task (for example, the method is applied to the field of industrial vision: scrap steel / cloth / steel / PCB (printed circuit board) / ceramic tile, etc., and the target detection task can be understood as scrap steel detection, cloth detection, steel detection, etc.), the data enhancement mode and the network structure can be understood as the data enhancement mode and the network structure of the target detection model, and the initial neural network model is the initial detection model; if the task to be processed is a classification task (such as commodity classification), the data enhancement mode and the network structure can be understood as the data enhancement mode and the network structure of the classification model, and the initial neural network model is the initial classification model; if the task to be processed is a text recognition task, the data enhancement mode and the network structure can be understood as the data enhancement mode and the network structure of the text recognition model, and the initial neural network model is the initial text recognition model.

[0064] The object training sample and the object verification sample are different when the data enhancement manner and the network structure are different during the synchronous search. For example, the object training sample and the object verification sample can be understood as the picture training sample and the picture verification sample when the data enhancement manner and the network structure are understood as the data enhancement manner and the network structure of the target detection model. When the data enhancement manner and the network structure are understood as the data enhancement manner and the network structure of the target detection model, the object training sample and the object verification sample can be understood as the text training sample and the text verification sample.

[0065] For ease of understanding, in the embodiments of the present specification, the initial neural network model is taken as the target detection model, and the object training sample and the object verification sample are taken as the picture training sample and the picture verification sample for detailed introduction.

[0066] The picture training sample and the picture verification sample can be obtained from any source and can be any type and any format of historical pictures.

[0067] In actual application, the picture training sample and the picture verification sample can also be pictures extracted from video frames obtained by an AR (augmented reality) device and / or a VR (virtual reality) device. The synchronous search of the data enhancement manner and the network structure is performed by using pictures obtained from three-dimensional video data to increase the richness of the samples and improve the recognition breadth and accuracy of the target detection model containing the data enhancement manner and the network structure. The specific implementation manner is as follows:

[0068] Before the determination of the object training sample, the object verification sample, and the initial neural network model, the method further includes:

[0069] Obtaining three-dimensional video data in an AR device and / or a VR device.

[0070] Extracting video frames in the three-dimensional video data to construct the picture training sample and the picture verification sample.

[0071] Step 204: obtaining a prediction result of the object training sample according to the initial data enhancement manner and the initial network structure that are differentiable in the initial neural network model.

[0072] Specifically, the obtaining of the prediction result of the object training sample according to the initial data enhancement manner and the initial network structure that are differentiable in the initial neural network model includes:

[0073] determining the initial data enhancement manner and the initial network structure that are differentiable in the initial neural network model, and an initial data enhancement atom set corresponding to the initial data enhancement manner and an initial network operation atom set corresponding to the initial network structure.

[0074] obtain an enhanced object training sample according to the object training sample and the initial data enhancement atom set;

[0075] input the enhanced object training sample into the initial network structure, combine the initial network operation atom set, and obtain a prediction result of the enhanced object training sample.

[0076] The initial neural network model is different, and the initial data enhancement mode and the initial network structure corresponding to the initial neural network model are also different. The initial data enhancement atom set corresponding to the initial data enhancement mode and the initial network operation atom set corresponding to the initial network structure are also different.

[0077] Still taking the initial neural network model as the target detection model as an example, in the case of the initial neural network model being the target detection model, the differentiable initial data enhancement mode and the initial network structure in the initial neural network model can be understood as the initial data enhancement mode and the initial network structure for target detection in a picture, that is, the initial data enhancement mode includes but is not limited to Color (color), Rotate (rotation), Flip (flip), and the initial data enhancement atom set corresponding to the initial data enhancement mode includes but is not limited to Color, Rotate, Flip, and the initial network structure includes backbone, neck, and head, wherein the initial network operation atom set corresponding to the backbone includes but is not limited to various sizes of convolution, pooling, and other initial network operation atoms, the initial network operation atom set corresponding to the neck includes various feature fusion modes, and the initial network operation atom set corresponding to the head includes various feature transformation modes.

[0078] After determining the differentiable initial data enhancement mode and the initial network structure in the initial neural network model, and the initial data enhancement atom set corresponding to the initial data enhancement mode and the initial network operation atom set corresponding to the initial network structure, the enhanced object training sample can be obtained according to the object training sample and the initial data enhancement atom set. Then, the prediction result of the enhanced object training sample can be obtained according to the enhanced object training sample, the initial network operation atom set, and the initial network structure.

[0079] In specific implementation, the search of the data enhancement mode can be modeled as Bernoulli distribution and category distribution, that is, the search of the data enhancement mode is modeled as the result of solving Bernoulli distribution and category distribution. In this way, the optimal data enhancement mode is searched for subsequent actual application, and the generalization ability of the neural network model using the data enhancement mode is improved. The specific implementation is as follows:

[0080] obtaining an enhanced object training sample according to the object training sample and the initial data augmentation atom set, comprising:

[0081] determining an initial data augmentation atom in the initial data augmentation atom set;

[0082] modeling a differentiable Bernoulli distribution and a classification distribution according to the initial data augmentation atom;

[0083] determining a target data augmentation atom from the initial data augmentation atom according to the Bernoulli distribution and the classification distribution;

[0084] performing data augmentation on the object training sample according to the target data augmentation atom to obtain an enhanced object training sample.

[0085] In actual application, in the training process of a neural network model, in order to improve the generalization ability of the model, various perturbations need to be performed on the input data (such as pictures), that is, data augmentation (for example, Color, Rotate, Flip, etc.), but the execution probability and the augmentation amplitude of each data augmentation manner and the combination of which data augmentation manners to select need to be obtained through search.

[0086] In specific implementation, a plurality of initial data augmentation atoms in an initial data augmentation atom set are first determined; then a differentiable Bernoulli distribution is modeled according to the execution probability and the augmentation amplitude of each initial data augmentation atom relative to an object training sample; meanwhile, a preset number (such as two) of initial data augmentation atoms are randomly selected in the initial data augmentation atom set to obtain an atom combination manner of each two as a group, and a differentiable classification distribution is modeled according to the atom combination manner; finally, a target data augmentation atom can be determined from the initial data augmentation atom according to the Bernoulli distribution and the classification distribution, and the target data augmentation atom can be understood as a data augmentation manner obtained through search.

[0087] Referring to Figure 3 , Figure 3 A search diagram of a data augmentation manner in a data processing method provided by one embodiment of the present specification is shown.

[0088] Figure 3 The initial data augmentation atom set includes A N initial data augmentation atoms OP (such as Color, Rotate, Flip, Crop, etc.) corresponding to A initial data augmentation manners.

[0089] For example Figure 3The small vertical box in the middle shows how the probability of each operation being executed and the magnitude of the enhancement are modeled as a Bernoulli distribution; as shown... Figure 3 The outermost large frame in the middle section shows a combination consisting of any K OPs selected from all OPs. These combinations form a classification distribution, where a typical value for K is 2. In this case, the Bernoulli distribution and the classification distribution are differentiable, and the probability and magnitude of each operation's execution become trainable parameters. The various combinations that form a classification distribution are also differentiable, and the execution probability (or confidence level) of each combination also becomes a differentiable parameter. That is, by training the probability of each operator's execution, the magnitude of the augmentation, and the probability of each combination during the network training process, suitable data augmentation methods can be searched.

[0090] Confidence level regarding OP combination methods:

[0091] That is to There are several combinations, each corresponding to a confidence level of α. i , i = 1...N

[0092]

[0093] Where s(x) represents a set of data augmentation operations performed on the input image X, and c i c represents the weights that augment this set of data. i It is obtained by sampling from the class distribution p(c|π), where π is derived from... get.

[0094] During training, α is optimized by backpropagation of the loss function values ​​between the predicted results obtained from the training images and the labeled images. i After training is completed, based on α i The top M combinations are selected as the actual OP combinations used.

[0095] The execution probability and augmentation magnitude of each OP are represented by a Bernoulli distribution during the search process:

[0096] s(x)=b·O(x;m)+(1-b)·x,b~Bernoulli(β)

[0097] Where β is the execution probability of the OP, and m is the enhancement magnitude.

[0098] During training, β and m are optimized by backpropagation of the loss function values ​​between the prediction results obtained from the training images and the annotations; the final β and m of each operation in each of the M combinations are the search results of the data augmentation method.

[0099] In actual applications, after modeling the differentiable Bernoulli distribution and the classification distribution, the OP combination with high confidence in the classification distribution can be selected according to the classification distribution and the Bernoulli distribution, and the execution probability and the enhancement amplitude of each OP in the OP combination can be determined according to the Bernoulli distribution; then, the object training sample is subjected to data enhancement according to the selected OP combination and the execution probability and the enhancement amplitude of each OP in the OP combination, to obtain an enhanced object training sample, i.e., an enhanced object training sample. As known from the above description, the initial values (i.e., initial execution probability and initial enhancement amplitude) of the initial data enhancement atom can be determined by initially modeling the differentiable Bernoulli distribution and the classification distribution. The specific implementation manners are as follows:

[0100] The modeling of the differentiable Bernoulli distribution and the classification distribution according to the initial data enhancement atom includes:

[0101] Modeling the differentiable Bernoulli distribution according to the initial execution probability and the initial enhancement amplitude of the initial data enhancement atom relative to the object training sample;

[0102] In the initial data enhancement atom set, a preset number of initial data enhancement atoms are randomly selected for combination to obtain a preset number of atomic combination manners relative to the object training sample, and a differentiable classification distribution is modeled according to the initial confidence of the atomic combination manner.

[0103] The preset number can be set according to actual applications, for example, the preset number can be set to 2 or 3, etc.

[0104] Specifically, the specific implementation manner of modeling the differentiable Bernoulli distribution and the classification distribution according to the initial data enhancement atom can refer to the detailed description of the above embodiments, which will not be repeated here.

[0105] After the enhanced object training sample is determined, the enhanced object training sample can be input into the initial network structure to obtain a prediction result of the enhanced object training sample in combination with the initial network operation atom set.

[0106] Taking the initial neural network model as a target detection model as an example, the initial network structure can include backbone / neck / head components; then, the specific implementation manner of determining the prediction result of the enhanced object training sample according to the initial network structure in combination with the corresponding initial network operation atom set is as follows:

[0107] The initial network structure is an initial network structure of the target detection model, and the initial network structure includes a first initial network component, a second initial network component, and a third initial network component.

[0108] Correspondingly, the inputting the enhanced object training sample into the initial network structure, in combination with the initial network operation atom set, to obtain a prediction result of the enhanced object training sample comprises:

[0109] The output result of the first initial network component is input into the second initial network component, in combination with a second initial network operation atom set corresponding to the second initial network component, to obtain an output result of the second initial network component.

[0110] The output result of the first initial network component is input into the second initial network component, in combination with a second initial network operation atom set corresponding to the second initial network component, to obtain an output result of the second initial network component.

[0111] The output result of the second initial network component is input into the third initial network component, in combination with a third initial network operation atom set corresponding to the third initial network component, to obtain an output result of the third initial network component.

[0112] According to the output result of the third initial network component, a prediction result of the enhanced object training sample is determined.

[0113] Specifically, in the case where the initial network structure is an initial network structure of a target detection model, the initial network structure comprises a backbone (a first initial network component) / a neck (a second initial network component) / a head (a third initial network component).

[0114] In actual application, the inputting of the enhanced object training sample into the backbone, in combination with a first initial network operation atom set corresponding to the backbone, can obtain an output result of the backbone; the inputting of the output result of the backbone into the neck, in combination with a second initial network operation atom set corresponding to the neck, can obtain an output result of the neck; the inputting of the output result of the neck into the head, in combination with a third initial network operation atom set corresponding to the head, can obtain a prediction result of the enhanced object training sample after the operation of the enhanced object training sample by the initial network operation atom sets corresponding to the initial network components; subsequently, a loss function can be obtained according to the prediction result and a label result (i.e. a label of the object training sample), so as to adjust the network parameters of the initial neural network model and obtain an accurate initial neural network model.

[0115] In practical implementation, when the first initial network component is a backbone, the backbone includes multiple network blocks, and each network block includes multiple processing layers. When the training samples for the augmented object are input into the backbone, they need to be processed through the initial network operation atoms corresponding to each processing layer of each network block in the backbone to obtain the backbone's output. The specific implementation method is as follows:

[0116] The first initial network component includes multiple network blocks, and each network block includes multiple processing layers;

[0117] Accordingly, the step of inputting the enhanced object training samples into the first initial network component, and combining them with the first initial network operation atom set corresponding to the first initial network component to obtain the output result of the first initial network component includes:

[0118] Determine the first initial network operation atom in the first initial network operation atom set corresponding to the first initial network component;

[0119] If it is determined that the network block does not have a previous network block and the current processing layer of the network block does not have a previous processing layer, the training sample of the augmentation object is input into the current processing layer of the network block, and the training sample of the augmentation object is operated on according to the first initial network operation atom to obtain the operation result of the current processing layer;

[0120] Based on the weights of the first initial network operation atoms, the operation results of the current processing layer are weighted and summed to obtain the sample feature map output by the current processing layer;

[0121] Based on the first initial network operation atom and the sample feature map output by the current processing layer, determine the sample feature map output by the next processing layer of the current processing layer;

[0122] Until all processing layers of all network blocks have been completed, the sample feature map of the enhanced object training sample output by the first initial network component is obtained.

[0123] In this context, if a network block does not have a preceding network block and its current processing layer does not have a preceding processing layer, it can be understood that the network block is the first network block of the backbone, and its current processing layer is the first processing layer of the first network block. The first initial network operation atoms in the first initial network operation atom set corresponding to the first initial network component include, but are not limited to, conv3x3 (3x3 convolution), conv5x5 (5x5 convolution), pooling, etc.

[0124] In practical applications, the first initial network component backbone is used to extract features of the enhanced object training sample, the backbone is composed of multiple network blocks, and each block includes multiple processing layers. Each processing layer needs to select a first initial network operation atom OP (conv3x3, conv5x5, pooling, etc.) to extract features of the enhanced object training sample. The optimal result is obtained by searching.

[0125] Specifically, first, a first initial network operation atom in a first initial network operation atom set corresponding to a first initial network component is determined. For a first processing layer of a first network block in the backbone, the enhanced object training sample is input into the first processing layer, and each first initial network operation atom is operated on the enhanced object training sample according to a preset (such as a conv3x3 operation, a conv5x5 operation, etc.), to obtain operation results of the first processing layer. The multiple operation results of the first processing layer are weighted and summed according to the weight of each first initial network operation atom, to obtain a sample feature map output by the first processing layer. The sample feature map output by the first processing layer is input into a next processing layer, and the sample feature map output by the first processing layer is operated on by each first initial network operation atom according to a preset (such as a conv3x3 operation, a conv5x5 operation, etc.), to obtain operation results of the next processing layer. The multiple operation results of the next processing layer are weighted and summed according to the weight of each first initial network operation atom, to obtain a sample feature map output by the next processing layer. In this way, a sample feature map output by a last processing layer of a last network block in the backbone is the sample feature map output by the backbone.

[0126] That is, the output of each previous processing layer is the input of a next processing layer, and the output of each next processing layer is determined according to the output of the previous processing layer and each first initial network operation atom. Through this training method, after training is completed, each processing layer can select the op with the maximum weight as the optimal op. The specific implementation manner is as follows:

[0127] The method further includes:

[0128] According to the first initial network operation atom, the sample feature map output by the current processing layer is operated, to obtain operation results of a next processing layer of the current processing layer.

[0129] According to the weight of the first initial network operation atom, the operation results of the next processing layer of the current processing layer are weighted and summed to obtain a sample feature map output by the next processing layer of the current processing layer.

[0130] As shown in formula 1, in the training process, each candidate OP (i.e., the first initial network operation atom) calculates the input X (an enhanced object training sample) to obtain a result using a trainable parameter a i The results of each candidate OP are weighted and summed as the output result of the current layer After the training is completed, the maximum a is taken i The corresponding OP is selected as the optimal OP.

[0131]

[0132] Wherein, x represents the feature map output by the previous processing layer, o(x) represents performing an operation on the input using a candidate OP, a i represents the weight of each OP, exp(x) represents an exponential function, represents the sample feature map output by the current layer.

[0133] According to formula 1, in the training process, the output of the current processing layer is the weighted sum of the operation results of each candidate OP on the input X; and in the optimization process, the verification set picture is used to optimize the parameter a i After multiple rounds of optimization through neural network back propagation, the optimized a i is obtained, the maximum a i corresponds to the optimal OP of the current layer.

[0134] Referring to Figure 4 , Figure 4 FIG. 1 shows a search schematic diagram of a network backbone in a data processing method according to an embodiment of the present specification.

[0135] Figure 4 The network backbone in FIG. 1 includes a plurality of network blocks Block (such as network block 1 to network block 6, i.e., Block1 to Block6), and each Block includes a plurality of processing layers.

[0136] Taking network block 2 as an example, the sample feature map extracted by network block 1 is input into network block 2, and each OP (such as Figure 4 conv3x3, conv5x5, Identity, Pool3x3 in FIG. 1, i.e., conv3x3, conv5x5, Identity, Pool3x3) in each processing layer in network block 2 is used to operate the input to obtain an operation result; and a trainable parameter a i is used (such as a in the figure1 -a N ) the operation results of each OP are weighted and summed as the output result of the current layer The maximum one is taken after the training is completed i The corresponding OP is selected as the optimal OP.

[0137] That is Figure 4 The input of the Input of the current processing layer of the network block 2 is the sample feature map output by Block1, and the output of the current processing layer of the Output

[0138] After obtaining the output result of the first initial network component, the output result of the first initial network component is input into the second initial network component, and the second initial network component corresponding to the second initial network operation atom set is combined to obtain a more accurate output result of the second initial network component relative to the enhanced object training sample. The specific implementation is as follows:

[0139] The output result of the first initial network component is input into the second initial network component, and the second initial network component corresponding to the second initial network operation atom set is combined to obtain the output result of the second initial network component, including:

[0140] Determine the second initial network operation atom in the second initial network operation atom set corresponding to the second initial network component;

[0141] The output result of the first initial network component is input into the second initial network component, and the second initial network operation atom is operated on the output result of the first initial network component to obtain the operation result of the second initial network component;

[0142] According to the weight of the second initial network operation atom, the operation result of the second initial network component is weighted and summed to obtain the output result of the second initial network component.

[0143] Among them, according to the above embodiment, the output result of the first initial network component can be understood as the sample feature map output by the last processing layer of the last network block in the network backbone.

[0144] Then, in the case of the second initial network component being the network neck, the output result of the network backbone is input into the network neck, and the network neck is operated according to the second initial network atom corresponding to the network neck to obtain the operation result of the network neck; and the operation result of the network neck is weighted and summed according to the weight of each second initial network operation atom, that is, the output result of the network neck is obtained.

[0145] Wherein, in the case of the second initial network component being a network neck, the second initial network operation atom can be understood as any fusion manner, such as TopDown (top-down), BottomUP (bottom-up), and the like.

[0146] Still in the above example, different scale feature maps of the enhanced object training sample are extracted through the network backbone, and in the network neck, the feature maps obtained by any Block in the network backbone can be selected for feature fusion, and the function of the network neck is to perform feature fusion on the selected feature maps. How to select the fusion manner also needs to be searched to obtain an optimal result.

[0147] Referring to Figure 5 , Figure 5 Fig. 6 shows a schematic diagram of the overall search space of the network neck in the data processing method provided by one embodiment of the present specification.

[0148] Figure 5 F1, F2 to FN in the figure represent feature maps, such as Figure 4 the feature maps output by any one Block of the network backbone in the figure.

[0149] Referring to Figure 6 , Figure 6 Fig. 7 shows a schematic diagram of the specific processing of the network neck according to different OPs in the data processing method provided by one embodiment of the present specification.

[0150] Figure 6 F1, F2 to FN in the figure represent feature maps, such as i-1 and F1, F2 to FN in the figure represent feature maps, such as i TopDown (top-down), BottomUP (bottom-up), and Identity (identity) respectively represent an OP (i.e., a second initial network operation atom), that is, each OP is a trainable parameter β, that is, β 1 -β N .

[0151] In combination with Figure 4 , different scale feature maps of the enhanced object training sample are extracted through the network backbone, and the feature maps obtained by Block3-Block6 in the network backbone are denoted as F1, that is, F1 is composed of four feature maps The network backbone's function is to fuse these four feature maps. The specific fusion method chosen also requires searching to obtain optimal results. In practical applications, the selection of which blocks' feature maps in the network backbone to fuse can be set according to the actual application. Generally, the semantic information extracted from the feature maps output by earlier blocks is not abstract enough, while the feature maps output by later blocks are very small, resulting in a small target. Therefore, it is necessary to select feature maps for fusion based on actual needs.

[0152] Specifically, Figure 6 China F i-1 To F i The process represents one fusion of these four feature maps, and each fusion is as follows: Figure 6 The TopDown, BottomUP, and other operations shown in the figure specify which fusion method to select in a single fusion, as shown in Formula 2, using 8 trainable parameters. i (i.e. 8) 1 -β N The weighted sum of the operation results of each operation relative to the feature map is used as the output result F after this fusion. i After training is complete, take the largest β. i The corresponding OP is the optimal fusion method.

[0153]

[0154] Among them, F i-1 Let β represent the sample feature map output by the previous layer, o(x) represent performing one operation on the input using one OP, and β i Let exp(x) represent the weight of each operation, and F represent the exponential function. i This represents the feature map output by the current layer.

[0155] Specifically, according to Formula 2, during training, the output of the current layer is a weighted sum of the operations performed on the input by each candidate operation. During optimization, validation set images are used to optimize the parameter β. i After multiple rounds of optimization via backpropagation in the God network, an optimized β can be obtained. i The largest β i The corresponding OP is the optimal fusion method for the current layer.

[0156] See Figure 7 , Figure 7 This diagram illustrates a feature fusion process using TopDown in a network neck within a data processing method provided in one embodiment of this specification.

[0157] Figure 7F i-1 The four feature maps in F i The specific processing process.

[0158] Specifically, F i-1 The four feature maps in F Assume that P1 is the feature map with the smallest resolution, and P4 is the feature map with the largest resolution, After a convolution layer, it becomes Similarly, After a convolution layer, it becomes Because the resolution of P1 is smaller than that of P2 (for example, the resolution of P1 is half the size of that of P2), P1 is up-sampled and added to P2 to obtain an output a1; and After a convolution layer, it becomes Because the resolution of P2 is smaller than that of P3 (for example, the resolution of P2 is half the size of that of P3), P2 is up-sampled and added to P3 to obtain an output a2; and After a convolution layer, it becomes Because the resolution of P3 is smaller than that of P4 (for example, the resolution of P3 is half the size of that of P4), P3 is up-sampled and added to P4 to obtain an output a3; and After a convolution layer, it becomes After up-sampling, the output of the third layer is added to a1 to obtain an output a2; and After a convolution layer, it becomes After up-sampling, the output of the fourth layer is added to a2 to obtain the final output of the fourth layer. After obtaining the output result of the second initial network component, the output result of the second initial network component is input into a third initial network component, a third initial network atom is combined, and an accurate prediction result of the third initial network component for the enhanced object training sample is obtained. The specific implementation manner is as follows: The output result of the third initial network component is obtained by inputting the output result of the second initial network component into the third initial network component and combining the third initial network operation atom set corresponding to the third initial network component, and the output result of the third initial network component is obtained.

[0159] The third initial network operation atom in the third initial network operation atom set corresponding to the third initial network component is determined;

[0160] The output result of the second initial network component is input into the third initial network component, the output result of the second initial network component is operated according to the third initial network operation atom, and the operation result of the third initial network component is obtained;

[0161] The third initial network operation atom in the third initial network operation atom set corresponding to the third initial network component is determined;

[0162] The output result of the second initial network component is input into the third initial network component, the output result of the second initial network component is operated according to the third initial network operation atom, and the operation result of the third initial network component is obtained;

[0163] ​​According to the weights of the third initial network operation atoms, the operation results of the third initial network components are weighted and summed to obtain an output result of the third initial network component.

[0164] In combination with the above embodiments, the output result of the second initial network component can be understood as the fusion feature map output by the network neck.

[0165] Then, in the case where the third initial network component is a network head, the output result of the network neck is input into the network neck, and the operation result of the network head is obtained according to the third initial network atom corresponding to the network head; and the output result of the network head is obtained by weighted summing the operation result of the network head according to the weight of each third initial network operation atom.

[0166] In the case where the third initial network component is a network head, the third initial network operation atom can be understood as an arbitrary feature transformation operation.

[0167] Still in the above example, after the network neck part performs feature fusion for N times, the fused feature F N is obtained. N After the network head part uses F N to perform multiple feature transformations of H1-H N , classification and regression for target detection are performed. The feature transformations of H1-H N use which OP (third initial network operation) can also be obtained by searching.

[0168] Referring to Figure 8 , Figure 8 , a schematic diagram of the overall search of the network head in a data processing method provided by one embodiment of the present specification is shown.

[0169] Figure 8 The feature F N in can be understood as a feature map after feature fusion by the network neck, and the H1 in to H N in N can be understood as the H N in can be understood as 1-N times of feature transformation of the feature F N .

[0170] Referring to Figure 9 , Figure 9 , a schematic diagram of specific processing of the network head according to different OPs in a data processing method provided by one embodiment of the present specification is shown.

[0171] Figure 9 The feature F N in and respectively represent a feature map after feature transformation, and conv3x3, conv5x5, identity, and Pool3x3 represent an OP (i.e., a second initial network operation atom), respectively, that is, each OP is a trainable parameter γ, that is, γ 1 -γ N .

[0172] Specifically, in the feature transformation process from to , the operation results of each OP with respect to are weighted and summed using a trainable parameter γ i as the output result after this fusion

[0173] As shown in formula 3, the operation results of each OP with respect to the feature map are weighted and summed using a trainable parameter γ i as the output result H i after this fusion, and the maximum γ i corresponding to the OP is selected as the optimal OP.

[0174]

[0175] where H i-1 represents the feature map output by the previous layer, o(x) represents an operation on the input using an OP, γ i represents the weight of each OP, exp(x) represents an exponential function, and H i represents the feature map output by the current layer.

[0176] The output result of the third initial network component can be understood as the prediction result of the enhanced object training sample.

[0177] Step 206: Adjust the network parameters of the initial neural network model according to the loss function constituted by the prediction result and the labeled result.

[0178] Step 208: Synchronously adjust the initial data enhancement manner and the initial network structure of the initial neural network model according to the object verification sample and the network parameters.

[0179] Specifically, the synchronous adjustment of the initial data enhancement manner and the initial network structure of the initial neural network model according to the object verification sample and the network parameters includes:

[0180] inputting the object verification sample into the initial neural network model adjusted according to the network parameters to obtain a prediction result of the object verification sample;

[0181] adjusting the initial data augmentation manner and the initial network structure of the initial neural network model according to a loss function constituted by the prediction result of the object prediction sample and a labeled result.

[0182] In specific implementation, the object verification sample is inputted into the initial neural network model adjusted according to the network parameters to obtain the prediction result of the object verification sample; and the initial data augmentation manner and the initial network structure of the initial network model are adjusted in a reverse propagation manner according to a loss function constituted by the prediction result of the object prediction sample and a labeled result, that is, the execution probability, the augmentation amplitude of the initial data augmentation atom corresponding to the initial data augmentation manner, and the network parameters (the initial network operation atom of each layer) in the initial network structure.

[0183] In actual application, each optimization of the synchronous search of the data augmentation and the network structure of the initial neural network model includes two steps. In the first step, the probability, the amplitude of the data augmentation, and the network structure parameter theta are fixed, K pictures (for example, K = 32) are taken from the training set, and the network parameters w are optimized according to a loss function constituted by the prediction result of each picture and a labeled result, that is, the network parameters are optimized while the network structure is fixed. In the second step, the network parameters w are fixed, K pictures (for example, K = 32) are taken from the verification set, and the probability, the amplitude of the data augmentation, and the network structure parameter theta are optimized according to a loss function constituted by the prediction result of each picture and a labeled result, that is, the probability, the amplitude of the data augmentation, and the network structure are optimized while the network parameters are fixed. The above two steps are repeatedly alternated until the network parameters w and the probability, the amplitude of the data augmentation, and the network structure parameter theta are all optimal, that is, the probability, the amplitude of the data augmentation, and the network structure parameter theta are the search results. The specific optimization algorithm can be a stochastic gradient descent method, and the probability, the amplitude of the data augmentation, and the network structure parameter theta (that is, the above-mentioned a, b, and g) are updated according to the gradient calculated in the reverse propagation process.

[0184] Step 210: determining a target data augmentation manner and a target network structure when the adjusted initial data augmentation manner and the initial network structure meet a preset ending condition.

[0185] The preset ending condition can be set according to actual application, for example, the preset ending condition can be understood as that the iteration meets a preset number, such as 100 times or 200 times, etc.

[0186] In actual applications, the object training sample and the object verification sample are used to iteratively optimize the data augmentation, the network structure, and the network parameter of the initial neural network model through the above embodiment, until the iteration number of the adjusted initial data augmentation and the initial network structure in the initial neural network model satisfies 100 times or 200 times or the like, and the iteration ends; then, according to the prediction result of the object verification sample, the accuracy of the initial neural network model including the adjusted initial data augmentation and the initial network structure, the preferred target data augmentation and the target network structure in the initial neural network model are determined. The specific implementation manner is as follows:

[0187] In the case where the adjusted initial data augmentation and the initial network structure satisfy the preset ending condition, the target data augmentation and the target network structure are determined, including:

[0188] In the case where the adjusted initial data augmentation and the initial network structure satisfy the preset ending condition, the target data augmentation and the target network structure are determined from the historical adjusted initial data augmentation and the initial network structure according to the prediction result of the object verification sample.

[0189] The historical adjusted initial data augmentation and the initial network structure can be understood as the historical iteration of the adjusted initial data augmentation and the initial network structure.

[0190] The data processing method provided by the embodiments of the present specification can obtain the complementary data augmentation and network structure on the corresponding data set at one time through the method of simultaneously searching the data augmentation and network structure of the neural network model (such as a detection model, a classification model, a segmentation model, etc.), thereby obtaining better search results, avoiding repeated training of network parameters, and greatly reducing the search time.

[0191] In actual applications, the data processing method provided by the embodiments of the present specification can be applied to the industrial vision field. In the industrial vision field, the amount of data that can be collected is small, and in order to obtain good flaw detection effect, it is necessary to perform appropriate data augmentation on the training data. In the application scenarios in the industrial vision field, the computer computing power usually used is not very high, and the size of the detection model and the computing power required will greatly affect the competitiveness of the product. Therefore, searching for a good data augmentation method and an efficient network structure of the target detection model is a common goal pursued. At the same time, a good data augmentation method and a good network structure are a complementary relationship, and a complementary result is obtained. The method for simultaneously searching for a data augmentation method and a network structure of a target detection model or other deep learning models provided by the embodiments of the present specification establishes a unified model for the data augmentation method and the network structure, so that a complementary data augmentation method and network structure on the corresponding data set are obtained at one time, and the industrial vision platform has a better user experience in the aspects of training data augmentation and model structure search.

[0192] Corresponding to the above method embodiments, the present specification also provides data processing device embodiments, Figure 10 The structure schematic diagram of a data processing device provided by one embodiment of the present specification is shown. As shown in the figure, Figure 10 The device comprises:

[0193] A first determination module 1002 is configured to determine object training samples, object verification samples, and an initial neural network model;

[0194] A result obtaining module 1004 is configured to obtain a prediction result of the object training samples according to the initial data augmentation method and the initial network structure that are differentiable in the initial neural network model;

[0195] A first adjustment module 1006 is configured to adjust network parameters of the initial neural network model according to a loss function constituted by the prediction result and a label result;

[0196] A second adjustment module 1008 is configured to synchronously adjust the initial data augmentation method and the initial network structure of the initial neural network model according to the object verification samples and the network parameters;

[0197] A second determination module 1010 is configured to determine a target data augmentation method and a target network structure in a case where the adjusted initial data augmentation method and the initial network structure meet a preset ending condition.

[0198] Optionally, the result obtaining module 1004 is further configured to:

[0199] determine the initial data augmentation manner and the initial network structure in the initial neural network model that are differentiable, and an initial data augmentation atom set corresponding to the initial data augmentation manner and an initial network operation atom set corresponding to the initial network structure;

[0200] obtain an augmented object training sample according to the object training sample and the initial data augmentation atom set;

[0201] input the augmented object training sample into the initial network structure, and combine the initial network operation atom set to obtain a prediction result of the augmented object training sample.

[0202] Optionally, the result obtaining module 1004 is further configured to:

[0203] determine an initial data augmentation atom in the initial data augmentation atom set;

[0204] model a differentiable Bernoulli distribution and a classification distribution according to the initial data augmentation atom;

[0205] determine a target data augmentation atom from the initial data augmentation atom according to the Bernoulli distribution and the classification distribution;

[0206] perform data augmentation on the object training sample according to the target data augmentation atom to obtain an augmented object training sample.

[0207] Optionally, the result obtaining module 1004 is further configured to:

[0208] model a differentiable Bernoulli distribution according to an initial execution probability and an initial augmentation amplitude of the initial data augmentation atom relative to the object training sample;

[0209] arbitrarily select a preset number of initial data augmentation atoms in the initial data augmentation atom set for combination to obtain a preset number of atom combination manners relative to the object training sample, and model a differentiable classification distribution according to an initial confidence of the atom combination manner.

[0210] Optionally, the initial network structure is an initial network structure of a target detection model, and the initial network structure includes a first initial network component, a second initial network component, and a third initial network component.

[0211] Correspondingly, the result obtaining module 1004 is further configured to:

[0212] input the augmented object training sample into the first initial network component, combine a first initial network operation atom set corresponding to the first initial network component, and obtain an output result of the first initial network component.

[0213] inputting the output result of the first initial network component into the second initial network component, and combining a second initial network operation atom set corresponding to the second initial network component to obtain an output result of the second initial network component;

[0214] inputting the output result of the second initial network component into the third initial network component, and combining a third initial network operation atom set corresponding to the third initial network component to obtain an output result of the third initial network component;

[0215] determining a prediction result of the augmented object training sample according to the output result of the third initial network component.

[0216] Optionally, the first initial network component includes a plurality of network blocks, and each network block includes a plurality of processing layers.

[0217] Correspondingly, the result obtaining module 1004 is further configured to:

[0218] determining a first initial network operation atom in a first initial network operation atom set corresponding to the first initial network component;

[0219] in a case where it is determined that the network block does not have a previous network block and the current processing layer of the network block does not have a previous processing layer, inputting the augmented object training sample into the current processing layer of the network block, and operating the augmented object training sample according to the first initial network operation atom to obtain an operation result of the current processing layer;

[0220] performing weighted summation on the operation result of the current processing layer according to a weight of the first initial network operation atom to obtain a sample feature map output by the current processing layer;

[0221] determining a sample feature map output by a next processing layer of the current processing layer according to the first initial network operation atom and the sample feature map output by the current processing layer;

[0222] until all processing layers of all network blocks are processed, a sample feature map of the augmented object training sample output by the first initial network component is obtained.

[0223] Optionally, the result obtaining module 1004 is further configured to:

[0224] operating the sample feature map output by the current processing layer according to the first initial network operation atom to obtain an operation result of a next processing layer of the current processing layer;

[0225] According to the weight of the first initial network operation atom, the operation results of a next processing layer of the current processing layer are weighted and summed to obtain a sample feature map output by the next processing layer of the current processing layer.

[0226] Optionally, the result obtaining module 1004 is further configured to:

[0227] determine a second initial network operation atom in a second initial network operation atom set corresponding to the second initial network component;

[0228] input the output result of the first initial network component into the second initial network component, and operate the output result of the first initial network component according to the second initial network operation atom to obtain an operation result of the second initial network component;

[0229] According to the weight of the second initial network operation atom, the operation result of the second initial network component is weighted and summed to obtain an output result of the second initial network component.

[0230] Optionally, the result obtaining module 1004 is further configured to:

[0231] determine a third initial network operation atom in a third initial network operation atom set corresponding to the third initial network component;

[0232] input the output result of the second initial network component into the third initial network component, and operate the output result of the second initial network component according to the third initial network operation atom to obtain an operation result of the third initial network component;

[0233] According to the weight of the third initial network operation atom, the operation result of the third initial network component is weighted and summed to obtain an output result of the third initial network component.

[0234] Optionally, the second adjusting module 1008 is further configured to:

[0235] input the object verification sample into the initial neural network model adjusted according to the network parameters to obtain a prediction result of the object verification sample;

[0236] synchronously adjust the initial data augmentation manner and the initial network structure of the initial neural network model according to a loss function constituted by the prediction result of the object prediction sample and the labeled result.

[0237] Optionally, the second determining module 1010 is further configured to:

[0238] In a case where the adjusted initial data augmentation manner and the initial network structure satisfy a preset ending condition, a target data augmentation manner and a target network structure are determined from the historical adjusted initial data augmentation manner and the initial network structure according to a prediction result of the object verification sample.

[0239] Optionally, the object training sample includes a picture training sample, and the object verification sample includes a picture verification sample.

[0240] The first initial network component includes a backbone, the second initial network component includes a neck, and the third initial network component includes a head.

[0241] Optionally, the apparatus further includes:

[0242] The sample acquisition module is configured to:

[0243] Acquire three-dimensional video data in an AR device and / or a VR device.

[0244] Extract a video frame in the three-dimensional video data, and construct the picture training sample and the picture verification sample.

[0245] The data processing apparatus provided by the embodiments of the present specification can obtain complementary data augmentation manners and network structures on a corresponding data set at one time through the method of simultaneously searching for data augmentation manners and network structures of a neural network model (such as a detection model, a classification model, a segmentation model, etc.), thereby obtaining better search results, avoiding repeated training of network parameters multiple times, and greatly reducing search time.

[0246] The above is a schematic scheme of the data processing apparatus of the present embodiment. It should be noted that the technical scheme of the data processing apparatus belongs to the same concept as the technical scheme of the data processing method described above, and the details of the technical scheme of the data processing apparatus that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0247] In combination with the above method embodiments, the data processing method provided in the above embodiments is applied to a scrap steel detection task to construct a scrap steel detection model. The scrap steel detection model can be used to accurately identify scrap steel in the future. The specific implementation manner is as follows:

[0248] A scrap steel detection model construction method includes:

[0249] Determine a scrap steel image training sample, a scrap steel image verification sample, and an initial scrap steel detection model.

[0250] According to the initial scrap detection model, the differentiable initial data enhancement manner, and the initial network structure, a prediction result of the scrap image training sample is obtained;

[0251] According to a loss function composed of the prediction result and a labeled result, network parameters of the initial scrap detection model are adjusted;

[0252] According to the scrap image verification sample and the network parameters, the initial data enhancement manner and the initial network structure of the initial scrap detection model are synchronously adjusted;

[0253] In a case where the adjusted initial data enhancement manner and the initial network structure satisfy a preset ending condition, a target data enhancement manner and a target network structure are determined;

[0254] According to the target data enhancement manner and the target network structure, a scrap detection model is constructed.

[0255] In a specific implementation, through the data processing method in the above embodiment, the training sample and the verification sample composed of the historical scrap image are used to synchronously search for the complementary target data enhancement manner and the target network structure, and finally the scrap detection model is constructed through the target data enhancement manner and the target network structure, so that the scrap detection model has a more optimal recognition effect in scrap detection, and user experience is improved.

[0256] Figure 11 A structural block diagram of a computing device 1100 according to one embodiment of the present specification is shown. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 through a bus 1130, and a database 1150 is used to save data.

[0257] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1060. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near-field communication (NFC) interface, and the like.

[0258] In one embodiment of the present specification, the above components of the computing device 1100 and Figure 11Other components not shown can also be connected to each other in the system, such as through a bus. It should be understood that Figure 11 The illustrated computing device structural block diagram is merely for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed.

[0259] The computing device 1100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. The computing device 1100 can also be a mobile or stationary server.

[0260] The processor 1120 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method or the scrap steel detection model construction method.

[0261] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device belongs to the same concept as the technical scheme of the data processing method or the scrap steel detection model construction method, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data processing method or the scrap steel detection model construction method.

[0262] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method or the scrap steel detection model construction method.

[0263] The above is a schematic scheme of the computer-readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium belongs to the same concept as the technical scheme of the data processing method or the scrap steel detection model construction method, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the data processing method or the scrap steel detection model construction method.

[0264] An embodiment of the present specification also provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the data processing method or the scrap steel detection model construction method.

[0265] The above is a schematic scheme of the computer program of the embodiment. It should be noted that the technical scheme of the computer program is the same as the technical scheme of the data processing method or the scrap steel detection model construction method, and the technical scheme of the computer program is not described in detail, and the description of the technical scheme of the data processing method or the scrap steel detection model construction method can be referred to.

[0266] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which they are recited in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0267] The computer instructions include computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the contents contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0268] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily all necessary for the embodiments of the present specification.

[0269] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0270] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. Alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and variations of the embodiments of the present specification can be made in light of the content of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: determining an object training sample, an object validation sample, and an initial neural network model, wherein the object training sample is a picture training sample or a text training sample, and the object validation sample is a picture validation sample or a text validation sample; obtaining a prediction result of the object training sample according to an initial data augmentation manner and an initial network structure in the initial neural network model, wherein the initial network structure comprises a plurality of components, each of the plurality of components of the initial network structure is decomposed into a set of initial network operation atoms that are differentiable, and the set of initial network operation atoms comprises different initial network operation atoms that are different in structure; adjusting network parameters of the initial neural network model according to a loss function constituted by the prediction result and a labeled result; synchronously adjusting the initial data augmentation manner and the initial network structure of the initial neural network model according to the object validation sample and the network parameters; and determining a target data augmentation manner and a target network structure when the adjusted initial data augmentation manner and the initial network structure satisfy a preset ending condition.

2. The data processing method of claim 1, wherein the obtaining of the prediction result of the object training sample according to the initial data augmentation manner and the initial network structure in the initial neural network model comprises: determining the initial data augmentation manner and the initial network structure in the initial neural network model, an initial data augmentation atom set corresponding to the initial data augmentation manner, and an initial network operation atom set corresponding to the initial network structure; obtaining an augmented object training sample according to the object training sample and the initial data augmentation atom set; inputting the augmented object training sample into the initial network structure and combining the initial network operation atom set to obtain a prediction result of the augmented object training sample.

3. The data processing method of claim 2, wherein the obtaining of the augmented object training sample according to the object training sample and the initial data augmentation atom set comprises: determining an initial data augmentation atom in the initial data augmentation atom set; modeling a differentiable Bernoulli distribution and a classification distribution according to the initial data augmentation atom; determining a target data augmentation atom from the initial data augmentation atom according to the Bernoulli distribution and the classification distribution; and performing data augmentation on the object training sample according to the target data augmentation atom to obtain the augmented object training sample.

4. The data processing method of claim 3, wherein the modeling of the differentiable Bernoulli distribution and the classification distribution according to the initial data augmentation atom comprises: modeling a differentiable Bernoulli distribution according to an initial execution probability and an initial augmentation amplitude of the initial data augmentation atom with respect to the object training sample. In the initial data enhancement atom set, a preset number of initial data enhancement atoms are selected and combined to obtain a preset number of atomic combination modes relative to the object training sample, and a differentiable classification distribution is modeled according to an initial confidence of the atomic combination mode.

5. The data processing method of claim 2, wherein the initial network structure is an initial network structure of an object detection model, and the initial network structure comprises a first initial network component, a second initial network component, and a third initial network component. Accordingly, the inputting the enhanced object training sample into the initial network structure and combining the initial network operation atom set to obtain a prediction result of the enhanced object training sample comprises: inputting the enhanced object training sample into the first initial network component, combining the first initial network operation atom set corresponding to the first initial network component, and obtaining an output result of the first initial network component; inputting the output result of the first initial network component into the second initial network component, combining the second initial network operation atom set corresponding to the second initial network component, and obtaining an output result of the second initial network component; inputting the output result of the second initial network component into the third initial network component, combining the third initial network operation atom set corresponding to the third initial network component, and obtaining an output result of the third initial network component; determining the prediction result of the enhanced object training sample according to the output result of the third initial network component.

6. The data processing method of claim 5, wherein the first initial network component comprises a plurality of network blocks, and each network block comprises a plurality of processing layers. Accordingly, the inputting the enhanced object training sample into the first initial network component and combining the first initial network operation atom set corresponding to the first initial network component to obtain an output result of the first initial network component comprises: determining a first initial network operation atom in the first initial network operation atom set corresponding to the first initial network component; in a case where it is determined that the network block does not have a previous network block and the current processing layer of the network block does not have a previous processing layer, inputting the enhanced object training sample into the current processing layer of the network block, and operating the enhanced object training sample according to the first initial network operation atom to obtain an operation result of the current processing layer; performing weighted summation on the operation result of the current processing layer according to the weight of the first initial network operation atom to obtain a sample feature map output by the current processing layer; determining a sample feature map output by a next processing layer of the current processing layer according to the first initial network operation atom and the sample feature map output by the current processing layer; until all processing layers of all network blocks are processed, obtaining a sample feature map of the enhanced object training sample output by the first initial network component.

7. The data processing method of claim 6, wherein the determining the sample feature map output by the next processing layer of the current processing layer according to the first initial network operation atom and the sample feature map output by the current processing layer comprises: performing operation on the sample feature map output by the current processing layer according to the first initial network operation atom to obtain an operation result of the next processing layer of the current processing layer; and performing weighted summation on the operation result of the next processing layer of the current processing layer according to the weight of the first initial network operation atom to obtain the sample feature map output by the next processing layer of the current processing layer.

8. The data processing method of claim 6, wherein the inputting the output result of the first initial network component into the second initial network component and combining the second initial network operation atom set corresponding to the second initial network component to obtain the output result of the second initial network component comprises: determining a second initial network operation atom in the second initial network operation atom set corresponding to the second initial network component; inputting the output result of the first initial network component into the second initial network component and performing operation on the output result of the first initial network component according to the second initial network operation atom to obtain an operation result of the second initial network component; and performing weighted summation on the operation result of the second initial network component according to the weight of the second initial network operation atom to obtain the output result of the second initial network component.

9. The data processing method of claim 6, wherein the inputting the output result of the second initial network component into the third initial network component and combining the third initial network operation atom set corresponding to the third initial network component to obtain the output result of the third initial network component comprises: determining a third initial network operation atom in the third initial network operation atom set corresponding to the third initial network component; inputting the output result of the second initial network component into the third initial network component and performing operation on the output result of the second initial network component according to the third initial network operation atom to obtain an operation result of the third initial network component; and performing weighted summation on the operation result of the third initial network component according to the weight of the third initial network operation atom to obtain the output result of the third initial network component.

10. The data processing method of claim 1, wherein the synchronously adjusting the initial data augmentation manner and the initial network structure of the initial neural network model according to the object verification sample and the network parameter comprises: inputting the object verification sample into the initial neural network model adjusted according to the network parameter to obtain a prediction result of the object verification sample; and synchronously adjusting the initial data augmentation manner and the initial network structure of the initial neural network model according to a loss function constituted by the prediction result of the object verification sample and a label result. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 11. The data processing method of claim 10, wherein in a case where the adjusted initial data augmentation manner and the initial network structure satisfy a preset ending condition, the target data augmentation manner and the target network structure are determined, comprising: in a case where the adjusted initial data augmentation manner and the initial network structure satisfy a preset ending condition, determining the target data augmentation manner and the target network structure from the historical adjusted initial data augmentation manner and the initial network structure according to the prediction result of the object validation sample.

12. The data processing method of claim 5, wherein the object training sample comprises a picture training sample, and the object validation sample comprises a picture validation sample; the first initial network component comprises a backbone, the second initial network component comprises a neck, and the third initial network component comprises a head.

13. The data processing method of claim 12, further comprising, before the object training sample, the object validation sample, and the initial neural network model are determined: obtaining three-dimensional video data in an AR device and / or a VR device; and extracting video frames in the three-dimensional video data to construct the picture training sample and the picture validation sample.

14. A scrap steel detection model construction method, comprising: determining a scrap steel image training sample, a scrap steel image validation sample, and an initial scrap steel detection model; obtaining a prediction result of the scrap steel image training sample according to a differentiable initial data augmentation manner and an initial network structure in the initial scrap steel detection model, wherein the initial network structure comprises a plurality of components, the plurality of components of the initial network structure are respectively decomposed into differentiable initial network operation atomic sets, and the differentiable initial network operation atomic sets comprise different initial network operation atoms with different initial network operation atom structures; adjusting network parameters of the initial scrap steel detection model according to a loss function constituted by the prediction result and a label result; synchronously adjusting an initial data augmentation manner and an initial network structure of the initial scrap steel detection model according to the scrap steel image validation sample and the network parameters; in a case where the adjusted initial data augmentation manner and the initial network structure satisfy a preset ending condition, determining a target data augmentation manner and a target network structure; and constructing a scrap steel detection model according to the target data augmentation manner and the target network structure. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images