Neural network model training method and device for image processing

Abstract

The present disclosure provides a neural network model training method for image processing, which can be applied to the field of artificial intelligence technology. The method includes: performing data enhancement on the source image to be processed, adding the same transparent color and background images of different styles to obtain an enhanced data set; using an edge detection algorithm to extract the edge position of the captured object of each image in the enhanced data set; using a morphological processing method to calculate the boundary area information corresponding to the edge position, and intercepting the captured object information of each image in the enhanced data set according to the boundary area information to form a training data set; inputting the training data set into a convolutional neural network model for training to obtain an image classification model. The present disclosure also provides a neural network model training device, equipment, storage medium and program product for image processing.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically to a neural network model training method, device, equipment and medium for image processing. Background Art

[0002] The booming development of mobile Internet has made it possible to provide users with a variety of services on portable devices. The application platform of technologies such as image processing has gradually expanded from PC to mobile. Sufficient computing resources, time resources and storage resources are the key to ensuring model performance. Therefore, the image processing model deployed on the mobile terminal must have the characteristics of small footprint, high accuracy and good real-time performance.

[0003] The typical deep convolutional network has a very large number of parameters and requires high computing resources, which is not suitable for single terminal devices (such as cameras and mobile phones). For example, 50% of all model parameters in AlexNet are concentrated in its first fully connected layer, and 90% of VGG model parameters are fully connected layer parameters. With the development of mobile Internet, in order to use neural networks on portable devices, it is necessary to reduce the amount of computation and parameters of the model, and lightweight neural networks represented by the MobileNet series came into being.

[0004] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technology: the current mainstream image classification model is difficult to meet the business needs of image processing on mobile terminals. On the one hand, convolutional neural network models such as ResNet-50 have high accuracy, but they often require more resources; on the other hand, the emergence of lightweight neural networks such as MobileNet-v2 has greatly reduced resource consumption, but its accuracy cannot meet business needs. Therefore, how to balance model performance (such as accuracy) and resource usage (such as storage, computing or time resources) needs to be solved urgently. Summary of the invention

[0005] In view of this, the present disclosure provides a neural network model training method, apparatus, device and medium for image processing.

[0006] One aspect of the present disclosure provides a neural network model training method for image processing, comprising: performing data enhancement on a source image to be processed, adding a background image of the same transparent color and different styles, and obtaining an enhanced data set; using an edge detection algorithm to extract the edge position of the captured object of each image in the enhanced data set; using a morphological processing method to calculate boundary area information corresponding to the edge position, and intercepting the captured object information of each image in the enhanced data set according to the boundary area information to form a training data set; inputting the training data set into a convolutional neural network model for training to obtain an image classification model.

[0007] According to an embodiment of the present disclosure, data enhancement includes at least one of affine transformation, color transformation, Gaussian noise or blur processing.

[0008] According to an embodiment of the present disclosure, the affine transformation includes at least one of a rotation transformation, a scaling transformation, a reflection transformation or a shear transformation, and the scaling transformation includes an isotropic scaling transformation or a non-isotropic scaling transformation.

[0009] According to an embodiment of the present disclosure, the edge detection algorithm includes an edge detection algorithm based on a trained overall nested edge detection network model.

[0010] According to an embodiment of the present disclosure, the morphological processing method includes an erosion operation, a dilation operation, or an opening and closing operation.

[0011] According to an embodiment of the present disclosure, after the step of intercepting the captured object information of each image in the enhanced data set according to the boundary area information, it also includes: setting the background area outside the boundary area of ​​each image to white.

[0012] According to an embodiment of the present disclosure, model training is applied to a cloud data center, and the method also includes: dividing the image classification model into a first part model and a second part model according to a preset model cutting point; and deploying the first part model and the second part model on a mobile terminal and an edge server, respectively.

[0013] According to an embodiment of the present disclosure, after the step of deploying the first partial model and the second partial model in the mobile terminal and the edge server respectively, it also includes: compressing and encoding the first partial model and the second partial model respectively.

[0014] According to an embodiment of the present disclosure, the convolutional neural network model includes a lightweight network model.

[0015] According to an embodiment of the present disclosure, the lightweight network model includes a MobileNet-v2 network model, and the MobileNet-v2 network model includes an input layer, multiple feature extraction layers and a fully connected layer.

[0016] According to an embodiment of the present disclosure, the input layer scales each image in the training dataset to a preset size.

[0017] According to an embodiment of the present disclosure, the MobileNet-v2 network model is improved in the following ways: the feature extraction layer parameters in the pre-trained MobileNet-v2 network model are used as the initialization parameters of the image classification model, and the image classification model is fine-tuned according to the actual business data set of image processing; and in the fully connected layer, the output category is modified to the preset number of image types, and the parameters of the fully connected layer are initialized using the Xavier method.

[0018] According to an embodiment of the present disclosure, the MobileNet-v2 network model quantizes weights using a post-training quantization method.

[0019] According to an embodiment of the present disclosure, the post-training quantization method includes: after the convolutional neural network model training is completed, the trained weights and activation functions are quantized from 32-bit floating point numbers to 8-bit integers; or the trained weights and activation functions are quantized from 32-bit floating point numbers to or 16-bit floating point numbers.

[0020] Another aspect of the present disclosure provides a neural network model training device for image processing, including: a data enhancement module, used to perform data enhancement on a source image to be processed, adding the same transparent color and background images of different styles to obtain an enhanced data set; an edge detection module, used to extract the edge position of the captured object of each image in the enhanced data set using an edge detection algorithm; a morphological processing module, used to calculate the boundary area information corresponding to the edge position using a morphological processing method, and intercept the captured object information of each image in the enhanced data set according to the boundary area information to form a training data set; a model training module, used to input the training data set into a convolutional neural network model for training to obtain an image classification model.

[0021] According to an embodiment of the present disclosure, the model training module also includes: a model cutting unit, which is used to divide the image classification model into a first part model and a second part model according to a preset model cutting point, and deploy the first part model and the second part model on the mobile terminal and the edge server respectively; and a model compression unit, which is used to compress and encode the first part model and the second part model respectively.

[0022] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned neural network model training method for image processing.

[0023] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the above-mentioned neural network model training method for image processing.

[0024] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement the above-mentioned neural network model training method for image processing when executed.

[0025] Compared with the prior art, the neural network model training method, device, equipment and medium for image processing provided by the present disclosure have at least the following beneficial effects:

[0026] (1) Improve the existing convolutional neural network model to save the computing, storage and wireless resources occupied by the model while ensuring image processing capabilities, so as to achieve efficient deployment of lightweight network models under resource-constrained conditions;

[0027] (2) The present invention ensures the classification accuracy of the image classification model through multi-level image preprocessing such as data enhancement, edge detection and morphological processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0029] Figure 1 The system architecture of the neural network model training method and device for image processing according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 2 A flowchart of a neural network model training method for image processing according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 3 The operation flow chart of the neural network model training method for image processing according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 4 A structural diagram schematically shows various operations of affine transformation according to an embodiment of the present disclosure;

[0033] Figure 5 A schematic diagram showing a structure of boundary region information obtained by morphological processing according to an embodiment of the present disclosure;

[0034] Figure 6 The structure diagram of the improved Mobilenet-v2 network model according to the embodiment of the present disclosure is schematically shown;

[0035] Figure 7 A flowchart of a neural network model training method for image processing according to another embodiment of the present disclosure is schematically shown;

[0036] Figure 8 A block diagram of a neural network model training device for image processing according to an embodiment of the present disclosure is schematically shown;

[0037] Fig. 9 A block diagram schematically shows a model training module according to another embodiment of the present disclosure; and

[0038] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0040] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0041] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0042] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0043] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some boxes or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable storage medium storing instructions, which can be used by an instruction execution system or used in conjunction with an instruction execution system.

[0044] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0045] The present disclosure provides a neural network model training method for image processing, which can be applied to the field of artificial intelligence technology. The method includes: performing data enhancement on the source image to be processed, adding the same transparent color and background images of different styles, and obtaining an enhanced data set; using an edge detection algorithm to extract the edge position of the captured object of each image in the enhanced data set; using a morphological processing method to calculate the boundary area information corresponding to the edge position, and intercepting the captured object information of each image in the enhanced data set according to the boundary area information to form a training data set; inputting the training data set into a convolutional neural network model for training to obtain an image classification model.

[0046] Figure 1 The system architecture 100 of the neural network model training method and device for image processing according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0047] like Figure 1 As shown, the system architecture according to this embodiment may include a terminal device 101, an edge server 102, and a cloud data center 103. The terminal device 101 and the edge server 102 may communicate with each other via a communication link, and the edge server 102 and the cloud data center 103 may also communicate with each other via a communication link. The communication link may include various connection types, such as wired and / or wireless communication links, etc.

[0048] The terminal device 101 may be any electronic device that supports web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. In particular, the terminal device 101 may include a mobile terminal device, and more specifically, the mobile terminal device may include a mobile Internet of Things device. The mobile Internet of Things device is an Internet of Things terminal based on a mobile handheld device, such as an autonomous vehicle, a smart home appliance, a smart access control, a video terminal device, etc.

[0049] The user can use the terminal device 101 to interact with the edge server 102 through the communication link to receive or send messages, etc. Similarly, the edge server 102 can also interact with the cloud data center 103 through the communication link to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only for example).

[0050] The cloud data center 103 may be a data center, a cloud computing center, or a supercomputing center, and has powerful computing and storage capabilities, and may provide various management or services, such as a background management server (only as an example) that supports websites browsed by users using the terminal device 101. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device 101 through the edge server 102.

[0051] It should be noted that the neural network model training method for image processing provided in the embodiment of the present disclosure can generally be executed by the cloud data center 103. Accordingly, the neural network model training device for image processing provided in the embodiment of the present disclosure can generally be set in the cloud data center 103.

[0052] Specifically, the image processing involved in the embodiments of the present disclosure may include a mobile foreign currency recognition service, a mobile garbage classification service, or a digital recognition service. For example, the mobile foreign currency recognition service is used to accurately and quickly automatically recognize the currency, amount, and real-time exchange rate of foreign currencies; the mobile garbage classification service can automatically identify recyclable garbage, non-recyclable garbage, kitchen waste, or hazardous waste for citizens; the digital recognition service can automatically recognize bank card numbers or express delivery numbers.

[0053] It should be understood that Figure 1 The number of terminal devices, edge servers, and cloud data centers in the embodiment is only for illustration. Any number of terminal devices, edge servers, and cloud data centers may be provided as required.

[0054] Figure 2The flowchart of the neural network model training method for image processing according to an embodiment of the present disclosure is schematically shown. Figure 3 The operational flow chart of the neural network model training method for image processing according to an embodiment of the present disclosure is schematically shown.

[0055] Combination Figure 3 ,right Figure 2 In the embodiment of the present disclosure, the neural network model training method for image processing may include operations S201 to S204.

[0056] In operation S201, data enhancement is performed on the source image to be processed, and the same transparent color and background images of different styles are added to obtain an enhanced data set.

[0057] In operation S202 , an edge detection algorithm is used to extract edge positions of captured objects of each image in the enhanced data set.

[0058] In operation S203, a morphological processing method is used to calculate boundary region information corresponding to the edge position, and captured object information of each image in the enhanced data set is intercepted according to the boundary region information to form a training data set.

[0059] In operation S204, the training data set is input into a convolutional neural network model for training to obtain an image classification model.

[0060] Through the embodiments of the present disclosure, the classification accuracy of the image classification model is guaranteed through multi-level image preprocessing such as data enhancement, edge detection and morphological processing. In addition, the present disclosure can reduce the computing resource and storage resource requirements of the convolutional neural network model while ensuring image processing performance.

[0061] In order to prevent underfitting and simulate the real image processing request or search situation of the mobile user, the present disclosure adopts a data enhancement method to expand the size of the data set. In the embodiment of the present disclosure, the data enhancement may include at least one of affine transformation, color transformation, Gaussian noise or blur processing. After the data enhancement operation, the same transparent color and different styles of background images are uniformly added to the expanded data set to facilitate the subsequent edge position extraction.

[0062] Specifically, the affine transformation may include at least one of a rotation transformation, a scaling transformation, a reflection transformation or a shear transformation, and the scaling transformation includes an isotropic scaling transformation or a non-isotropic scaling transformation. The affine transformation includes the above rotation transformation, scaling transformation, reflection transformation or shear transformation, and a combination of any order or number of these transformations.

[0063] For ease of understanding, let the input image matrix be represented by P, the affine transformation matrix be represented by R, and the output matrix be represented by Q. Let the rotation angle be θ, and the horizontal and vertical translation displacements be t x and t y The scaling factor is represented by s, and the affine transformation process can be characterized as follows:

[0064]

[0065] Figure 4 The structure diagram of various operations of affine transformation according to an embodiment of the present disclosure is schematically shown.

[0066] like Figure 4 As shown, translation and rotation are conventional image transformations, and scaling can be further divided into isotropic scaling or non-isotropic scaling, where the scaling coefficients of each coordinate axis are the same, which is isotropic scaling, and other scaling is non-isotropic scaling. Shearing is to translate all points proportionally along a specified direction.

[0067] In addition, color transformation refers to the transformation of image brightness, saturation and contrast, and its implementation principle is to process the RGB three-channel data of the input image. Gaussian noise refers to the random superposition of noise data that satisfies the Gaussian distribution on the input image. Blurring is achieved by using a low-pass filter.

[0068] After adding background images of different styles to the augmented dataset, edge detection is continued to calculate the edge coordinates of useful content in the image, that is, to extract the objects to be classified from the impure background.

[0069] In the disclosed embodiment, the edge detection algorithm includes an edge detection algorithm based on a trained Holistically-Nested Edge Detection (HED) network model. It is understandable that the classic edge detection algorithm can be implemented by candy operators such as the cv2.Canny function and the cv2.findContours function in OpenCV, but many threshold parameters need to be set, and it cannot work properly in real mobile image processing scenarios. The present disclosure uses a trained HED network model to construct an edge detector that can work properly in general scenarios.

[0070] Through the embodiments of the present disclosure, for detecting edges and object boundaries in the field of image processing, the trained HED network model is used to implement end-to-end training and prediction based on the entire image, while completing multi-scale feature learning.

[0071] In order to further accurately enhance the boundary position of the captured object in each image in the dataset, the morphological processing method is used to make the edge contour of the captured object clearer.

[0072] In the disclosed embodiment, the morphological processing method includes an erosion operation, a dilation operation, or an opening and closing operation.

[0073] The erosion operation is to thin and shrink the target object in the input image, and the thinning procedure is controlled by the structural element. The dilation operation is to thicken or lengthen the input image, and the degree of thickening or lengthening is controlled by the set of structural elements.

[0074] Opening and closing operations are different combinations of erosion and expansion operations, including opening and closing operations. For example, the opening operation can be an erosion operation first and then an expansion operation, and the closing operation can be an expansion operation first and then an erosion operation.

[0075] Figure 5 The structure diagram of the boundary area information obtained by morphological processing according to the embodiment of the present disclosure is schematically shown.

[0076] like Figure 5 As shown, the captured object in the image may be, for example, a banknote, and the boundary area where the banknote is located is the white part in the figure. The boundary area is clear. It can be seen that the embodiment of the present disclosure can accurately locate the frame of the captured object in an impure background through multi-level image preprocessing such as data enhancement, edge detection and morphological processing, thereby ensuring the quality of image preprocessing.

[0077] Furthermore, after the step of intercepting the captured object information of each image in the enhanced data set according to the boundary area information, it also includes: setting the background area outside the boundary area of ​​each image to white. Thus, a uniform background is set outside the boundary area where the captured object is located, so as to facilitate the regular and uniform input to the subsequent convolutional neural network model for image classification.

[0078] In the disclosed embodiment, the convolutional neural network model includes a lightweight network model.

[0079] It is understandable that the computing power and storage space of mobile terminals are often limited, and excessive time consumption will seriously reduce the customer experience. Therefore, directly deploying convolutional neural networks on mobile devices with limited resources often cannot meet business needs. The present disclosure uses a lightweight network model to train the model in the cloud, which can save the computing, storage and wireless resources occupied by the model while ensuring the image processing capability, so as to realize the efficient deployment of lightweight network models under resource-constrained conditions.

[0080] Furthermore, in order to better balance the model's resource occupation and image processing capabilities, the lightweight network model may include a MobileNet-v2 network model, which includes an input layer, multiple feature extraction layers, and a fully connected layer.

[0081] The input layer scales each image in the training dataset to a preset size. To meet the image processing requirements of the MobileNet-v2 network model, the preset size can be 224×224×3.

[0082] Multiple feature extraction layers include convolutional layers, pooling layers, activation functions, batch normalization and other operations, which are used to complete the mapping of raw data to hidden layer feature space. The fully connected layer acts as a classifier, realizing the mapping of the "distributed feature representation" learned by the feature extraction layer to the sample label space. In actual use, the fully connected layer can be implemented by a convolution operation with a convolution kernel of 1×1.

[0083] In the disclosed embodiment, the MobileNet-v2 network model is improved in the following ways:

[0084] Use the feature extraction layer parameters in the pre-trained MobileNet-v2 network model as the initialization parameters of the image classification model, and fine-tune the image classification model based on the actual business data set of image processing; and

[0085] In the fully connected layer, the output category is modified to the preset number of image types, and the Xavier method is used to initialize the parameters of the fully connected layer.

[0086] It can be understood that the Xavier method can make the input and output obey the same distribution as much as possible, in order to prevent the output value of the ReLu activation function of the subsequent neural network layer from tending to 0.

[0087] Figure 6 The structure diagram of the improved Mobilenet-v2 network model according to an embodiment of the present disclosure is schematically shown.

[0088] Table 1 is a structural table of the improved Mobilenet-v2 network model according to an embodiment of the present disclosure. Figure 6 The main features of the improved Mobilenet-v2 network model are the inverted residual and the linear bottleneck. The inverted residual is mainly used to increase the extraction of image features to improve accuracy, while the linear bottleneck is mainly used to avoid information loss of nonlinear functions.

[0089] In order to make the RuLU activation function have good numerical resolution even when the mobile device has low precision, the activation range of the ReLU activation function is limited. Experiments have shown that limiting the output of ReLU to 6 is the best effect. The function after this restriction is renamed ReLU6.

[0090] In Table 1, t refers to the dimension expanded in the residual block; c is the dimension of the output, which can also be used to determine the number of convolution kernels; s is the step size of the convolution, which can be used to determine the multiple of downsampling; n is the number of times the current row operation is repeatedly applied.

[0091] Table 1

[0092] enter Calculator t c n s <![CDATA[224 2 ×3]]> Conv2d - 32 1 2 <![CDATA[112 2 ×32]]> Bottleneck 1 16 1 1 <![CDATA[112 2 ×16]]> Bottleneck 6 24 2 2 <![CDATA[56 2 ×24]]> Bottleneck 6 32 3 2 <![CDATA[28 2 ×32]]> Bottleneck 6 64 4 2 <![CDATA[14 2 ×64]]> Bottleneck 6 96 3 1 <![CDATA[14 2 ×96]]> Bottleneck 6 160 3 2 <![CDATA[7 2 ×160]]> Bottleneck 6 320 1 1 <![CDATA[7 2 ×320]]> Conv2d1×1 6 1280 1 1 <![CDATA[7 2 ×1280]]> Avgpool 7×7 - - 1 - 1×1×1280 Conv2d 1×1 - K - -

[0093] Combining Table 1 and Figure 6 , the output classification value K of the improved Mobilenet-v2 network model, that is, the number of preset image types.

[0094] Network quantization refers to the process of representing a network discretely into low-precision numerical points, thereby compressing the original network by reducing the number of bits required to represent each weight. In the disclosed embodiment, the MobileNet-v2 network model uses post-training quantization to quantize weights.

[0095] Specifically, the post-training quantization method includes: after the convolutional neural network model training is completed, the trained weights and activation functions are quantized from 32-bit floating point numbers to 8-bit integers; or the trained weights and activation functions are quantized from 32-bit floating point numbers to 16-bit floating point numbers.

[0096] Since the weight parameters and activation function output values ​​in the Mobilenet-v2 network model are 32-bit floating point numbers by default, this disclosure uses post-training quantization, that is, floating point numbers are still used during the training process, and weights are quantized after training is completed, quantizing weights and activation functions from 32-bit floating point numbers to 8-bit integers or 16-bit floating point numbers.

[0097] Through the embodiments of the present disclosure, the image classification model is improved based on the lightweight Mobilenet-v2 network model, and quantization compression is performed after training is completed to reduce the number of parameters and computational complexity.

[0098] Figure 7 A flowchart of a neural network model training method for image processing according to another embodiment of the present disclosure is schematically shown.

[0099] like Figure 7 For the sake of simplicity, the Figure 2The same features as those of the method shown in FIG. 1 are not described in detail. The following only describes the features different from the above-mentioned embodiments. That is, in another embodiment of the present disclosure, in the above-mentioned Figure 2 The method shown may further include additional operations S710 to S720.

[0100] In operation S710, the image classification model is divided into a first partial model and a second partial model according to a preset model cutting point.

[0101] In operation S720, the first partial model and the second partial model are deployed on the mobile terminal and the edge server respectively.

[0102] Furthermore, the terminal device includes a mobile terminal device, and the mobile terminal device includes a mobile Internet of Things device.

[0103] Through another embodiment of the present disclosure, the available resources in the three-level hierarchical network architecture of terminal device-edge server-cloud data center are fully utilized to save the computing, storage and wireless resources occupied by the model while ensuring the service quality, thereby realizing efficient deployment of the model on the mobile terminal under resource-constrained conditions.

[0104] Furthermore, in another embodiment of the present disclosure, after the step of deploying the first part model and the second part model in the mobile terminal and the edge server respectively, it may also include: compressing and encoding the first part model and the second part model respectively. Thus, after compression encoding, the first part model and the second part model are respectively deployed in the mobile terminal and the edge server, further optimizing the convolutional neural network model.

[0105] Based on the above-mentioned neural network model training method for image processing, the present disclosure also provides a neural network model training device for image processing. Figure 8 and Fig. 9 The device is described in detail.

[0106] Figure 8 A block diagram of a neural network model training device for image processing according to an embodiment of the present disclosure is schematically shown.

[0107] like Figure 8 As shown, the neural network model training device 800 for image processing may include a data enhancement module 810, an edge detection module 820, a morphological processing module 830 and a model training module 840.

[0108] The data enhancement module 810 is used to perform data enhancement on the source image to be processed, adding the same transparent color and background images of different styles to obtain an enhanced data set.

[0109] The edge detection module 820 is used to extract the edge position of the captured object of each image in the enhanced data set by using an edge detection algorithm.

[0110] The morphological processing module 830 is used to calculate the boundary area information corresponding to the edge position using a morphological processing method, and to intercept the captured object information of each image in the enhanced data set according to the boundary area information to form a training data set.

[0111] The model training module 840 is used to input the training data set into the convolutional neural network model for training to obtain an image classification model.

[0112] Furthermore, in another embodiment of the present disclosure, the model training module 840 may include a model cutting unit 910 and a model compression unit 920 .

[0113] The model cutting unit 910 is used to cut the image classification model into a first part model and a second part model according to a preset model cutting point, and deploy the first part model and the second part model on the mobile terminal and the edge server respectively.

[0114] The model compression unit 920 is used to compress and encode the first part of the model and the second part of the model respectively.

[0115] It should be noted that the device part in the embodiments of the present disclosure corresponds to the method part in the embodiments of the present disclosure. The description of the neural network model training device part for image processing specifically refers to the neural network model training method part for image processing, which will not be repeated here.

[0116] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0117] For example, any multiple of the data enhancement module 810, the edge detection module 820, the morphological processing module 830, the model training module 840, the model cutting unit 910, and the model compression unit 920 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the data enhancement module 810, the edge detection module 820, the morphological processing module 830, the model training module 840, the model cutting unit 910 and the model compression unit 920 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the data enhancement module 810, the edge detection module 820, the morphological processing module 830, the model training module 840, the model cutting unit 910 and the model compression unit 920 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0118] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Fig.10 As shown, the electronic device 1000 includes a processor 1010 and a computer-readable storage medium 1020. The electronic device 1000 can execute the neural network model training method for image processing according to an embodiment of the present disclosure.

[0120] Specifically, the processor 1010 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1010 may also include an onboard memory for cache purposes. The processor 1010 may be a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0121] The computer-readable storage medium 1020 may be, for example, a non-volatile computer-readable storage medium, and specific examples include but are not limited to: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a memory, such as a random access memory (RAM) or a flash memory; and the like.

[0122] The computer-readable storage medium 1020 may include a computer program 1021 , which may include code / computer-executable instructions that, when executed by the processor 1010 , cause the processor 1010 to perform a method according to an embodiment of the present disclosure or any variation thereof.

[0123] The computer program 1021 may be configured to have, for example, a computer program code including a computer program module. For example, in an example embodiment, the code in the computer program 1021 may include one or more program modules, for example, 1021A, module 1021B, ... It should be noted that the division method and number of modules are not fixed, and those skilled in the art may use appropriate program modules or program module combinations according to actual conditions, and when these program module combinations are executed by the processor 1010, the processor 1010 may execute the method according to the embodiment of the present disclosure or any variation thereof.

[0124] According to an embodiment of the present disclosure, at least one of the data enhancement module 810, the edge detection module 820, the morphological processing module 830, the model training module 840, the model cutting unit 910, and the model compression unit 920 may be implemented as a reference. Fig.10 The computer program modules described herein, when executed by the processor 1010, can implement the corresponding operations described above.

[0125] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the neural network model training method for image processing according to the embodiment of the present disclosure is implemented.

[0126] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0127] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the neural network model training method for image processing provided by the embodiment of the present disclosure.

[0128] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0129] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0131] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A neural network model training method for image processing, comprising: Perform data augmentation on the source image to be processed, add the same transparent color and background images of different styles, and obtain an augmented data set; Extracting edge positions of captured objects in each image in the enhanced dataset using an edge detection algorithm; Calculate the boundary area information corresponding to the edge position using a morphological processing method, and intercept the captured object information of each image in the enhanced data set according to the boundary area information to form a training data set; Inputting the training data set into a convolutional neural network model for training to obtain an image classification model; The model training is applied to a cloud data center, and the method further includes: According to a preset model cutting point, the image classification model is divided into a first part model and a second part model; Deploy the first part model and the second part model on the mobile terminal and the edge server respectively; The convolutional neural network model includes a lightweight network model; The lightweight network model includes a MobileNet-v2 network model, and the MobileNet-v2 network model includes an input layer, multiple feature extraction layers and a fully connected layer; The MobileNet-v2 network model is improved in the following ways: Using the feature extraction layer parameters in the pre-trained MobileNet-v2 network model as the initialization parameters of the image classification model, and fine-tuning the image classification model according to the actual business data set of image processing; and In the fully connected layer, the output category is modified to a preset number of image categories, and the parameters of the fully connected layer are initialized using the Xavier method.

2. The method according to claim 1, wherein: The data enhancement includes at least one of affine transformation, color transformation, Gaussian noise or blur processing.

3. The method according to claim 2, wherein: The affine transformation includes at least one of a rotation transformation, a scaling transformation, a reflection transformation or a shear transformation, and the scaling transformation includes an isotropic scaling transformation or a non-isotropic scaling transformation.

4. The method according to claim 1, wherein: The edge detection algorithm includes an edge detection algorithm based on a trained overall nested edge detection network model.

5. The method according to claim 1, wherein: The morphological processing method includes a corrosion operation, a dilation operation or an opening and closing operation.

6. The method according to claim 1, wherein: After the step of intercepting the captured object information of each image in the enhanced data set according to the boundary area information, the method further includes: The background area outside the boundary area of ​​each image is set to white.

7. The method according to claim 1, wherein: After the step of deploying the first part model and the second part model in the mobile terminal and the edge server respectively, the method further includes: The first part model and the second part model are compressed and encoded respectively.

8. The method according to claim 1, wherein: The input layer scales each image in the training dataset to a preset size.

9. The method according to claim 1, wherein: The MobileNet-v2 network model uses a post-training quantization method to quantize weights.

10. The method according to claim 9, wherein: The post-training quantization method includes: After the convolutional neural network model is trained, quantizing the trained weights and activation functions from 32-bit floating point numbers to 8-bit integers; or The trained weights and activation functions are quantized from 32-bit floating point numbers to 16-bit floating point numbers.

11. A neural network model training device for image processing, comprising: The data enhancement module is used to enhance the source image to be processed, add the same transparent color and background images of different styles, and obtain an enhanced data set; An edge detection module, configured to extract edge positions of captured objects of each image in the enhanced data set using an edge detection algorithm; A morphological processing module, configured to calculate boundary region information corresponding to the edge position using a morphological processing method, and intercept captured object information of each image in the enhanced data set according to the boundary region information to form a training data set; A model training module, used to input the training data set into a convolutional neural network model for training to obtain an image classification model; The model training module further includes: a model cutting unit, which is used to cut the image classification model into a first part model and a second part model according to a preset model cutting point, and deploy the first part model and the second part model on a mobile terminal and an edge server respectively; and a model compression unit, which is used to compress and encode the first part model and the second part model respectively; The convolutional neural network model includes a lightweight network model; The lightweight network model includes a MobileNet-v2 network model, and the MobileNet-v2 network model includes an input layer, multiple feature extraction layers and a fully connected layer; The MobileNet-v2 network model is improved in the following ways: Using the feature extraction layer parameters in the pre-trained MobileNet-v2 network model as the initialization parameters of the image classification model, and fine-tuning the image classification model according to the actual business data set of image processing; and In the fully connected layer, the output category is modified to a preset number of image categories, and the parameters of the fully connected layer are initialized using the Xavier method.

12. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 10 when executed.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images