A fast and accurate image detection and semantic classification system and method

Abstract

The present invention discloses a fast and accurate image detection and semantic classification system and method in the field of image detection and semantic classification technology, including: reading an image and image text information, predicting the coordinates of the upper left corner point and the lower right corner point of the image; selecting the range of the target image according to the coordinate values ​​of the upper left corner point and the lower right corner point of the image; reading the Bbox value in the image text information, analyzing the coordinate values ​​of each image in the image; judging whether the target image is in the selected prediction rectangular frame within the range of the target image based on the coordinate values ​​of each image in the image, wherein when the features in the target image exceeding a specified proportion are all located in the prediction rectangular frame, they are divided into valid prediction rectangular frames. The present invention can be applied to the processing of images, extracting image features, and semantically classifying a large number of images, improving the detection efficiency while improving the accuracy of the processing results.

Description

Technical Field

[0001] The invention relates to a fast and accurate picture detection and semantic classification system and method, belonging to the technical field of picture detection and semantic classification. Background Art

[0002] With the rapid development of mobile Internet, smart phones and social networks, a huge amount of picture information has been brought. According to the 2021 Mobile Internet Industry Data Research Report, WeChat sends 6.7 billion pictures every day, and mobile Internet users use their mobile phones for about 6.7 hours a day, browsing pictures and short video apps for about 50%. Pictures have become the main medium for Internet information exchange. When information is recorded in text, we can easily find the required content and edit it arbitrarily through keyword search, but when information is recorded in pictures, we cannot retrieve the content in the pictures, which affects our efficiency in finding key content from the pictures. Pictures bring us a quick way to record and share information, but reduce our information retrieval efficiency. In this environment, computer image detection technology is particularly important. At present, the image detection method based on deep learning mainly describes the position of the image in the picture by circling the prediction rectangle to calibrate the target object. However, when two images are close or overlapped, it will cause confusion in matching, the accuracy will drop significantly, and the time required for optimization will increase. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a fast and accurate image detection and semantic classification system and method, which can accurately determine the position of an image in a picture, reduce the time for searching for image information, and at the same time use a neural network for iterative optimization to significantly reduce errors.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0005] In a first aspect, the present invention provides a fast and accurate image detection and semantic classification method, comprising:

[0006] Read the image and its text information, and predict the coordinates of the upper left corner and lower right corner of the image;

[0007] According to the coordinate values ​​of the upper left corner and the lower right corner of the image, select the range of the target image;

[0008] Read the Bbox value in the image text information and analyze the coordinate values ​​of each image in the image;

[0009] Based on the coordinate values ​​of each image in the image, determine whether the target image is in the selected prediction rectangular frame within the range of the target image, wherein when all features exceeding a specified proportion in the target image are located in the prediction rectangular frame, it is divided into a valid prediction rectangular frame;

[0010] The VGG19 neural network model is used to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size to obtain a multi-dimensional feature vector;

[0011] Regressing the multi-dimensional feature vector into corresponding picture types through the softmax function in the neural network model, and obtaining the predicted probability of belonging to each picture type;

[0012] Constructing a first loss function and a second loss function for the upper left corner point and the lower right corner point according to the Bbox value of the picture text information in each of the picture types;

[0013] The recognition result of the effective prediction rectangle is optimized according to the first loss function and the second loss function in sequence, and then the RMSprop optimizer is selected to reduce the error value;

[0014] Construct training steps and test steps at the same time, set the number of training and optimization times, and iterate;

[0015] The predicted probability obtained after iterative optimization is compared with the set threshold, and the predicted probability greater than the set threshold is output as the result.

[0016] Furthermore, if all features of the target image are located in the prediction rectangular frame, it is divided into the best prediction rectangular frame; if features of the target image exceeding a specified proportion are not located in the prediction rectangular frame or all features are not in the prediction rectangular frame, it is divided into an invalid prediction rectangular frame.

[0017] Furthermore, the predicted rectangular frame is constructed by calling the ImageDraw module to directly read the Bbox value in the picture text information, and using the retangle module to accurately read the position of the image in the picture according to the Bbox value.

[0018] Furthermore, the VGG19 neural network model:

[0019] Convolution is performed through the Conv2d module, which applies a two-dimensional convolution on the input signal composed of multiple input planes. The expression is:

[0020]

[0021] Among them, * represents the 2D cross-correlation operator, out represents the output value, and input represents the input value. Indicates the number of output channels, bias indicates the bias value, weight indicates the weight, N i represents the number of images processed in the minimum batch size of the i-th image, C m Indicates the number of channels of the mth image, k indicates the convolution kernel size, and must be a positive integer;

[0022] Max pooling is performed through the MaxPool2d module, which applies a 2D max pool on an input signal composed of several input planes. The expression is:

[0023]

[0024] input(N i ,C i ,stride[0]×h+m,stride[1]×w+n)

[0025] N i represents the number of images processed in the minimum batch size of the i-th image, C i Indicates the number of channels of the i-th image, h indicates the height of the input plane in pixels; w indicates the width of the input plane in pixels; m and n indicate the number of images, kH is the height of the pooling kernel, and kW is the width of the pooling kernel. If the padding is non-zero, the input is implicitly padded with negative infinity on both sides, and stride indicates the step size of each step of the image during convolution;

[0026] The continuous grayscale range is flattened into a tensor through the Flatten module;

[0027] Linearization is performed through the Linear module to apply a linear transformation to the input data. The expression is:

[0028] Y=XA T +B

[0029] Y represents the output value, X represents the input value, A is the input matrix, T represents the transpose of the matrix, and B represents the deviation value.

[0030] Furthermore, the first loss function is CrossEntropyLoss, which is expressed as:

[0031] R(x,y)=L={R1,.R n ..,R N} T

[0032] {yn≠ignore index value}

[0033] In the formula, x is the input value, y is the target value, L represents the entire matrix set, C is the number of classes, Rn represents the matrix when the number is n, N is the number of images to be processed in the minimum batch, T represents the transposition of the matrix, yn represents the nth target value, and w yn represents the weight of the nth target value, x n,yn The input value when the number is n and the weight of the nth target value; x n,c Indicates the input value when the number is n and the type of image is C;

[0034] When given a value for the ignore index, the unreduced loss is described as:

[0035]

[0036] The above formula takes the average value, and the following formula takes the sum;

[0037] Probability of each class: Applicable when more than one class label is required for each mini-batch, the unreduced loss is described as:

[0038] R(x,y)=L={R1,.R n ..,R N} T

[0039]

[0040] In the formula, x is the input value, y is the target value, C is the number of classes, and w c represents the weight when the number of classes is C, N is the minimum size of the batch processed image, y n,c Indicates the target value when the number of classes is C;

[0041] The second loss function is MSELoss, and the expression is:

[0042] R(x,y)=L={R1,.R n .., R N} T

[0043] R n =(x n -y n ) 2

[0044] Among them, x n Indicates the input value when the quantity is n, y n Represents the output value when the number is n, and L represents the entire set of matrices.

[0045] Furthermore, the softmax function determines the similarity between two images by comparing the multi-dimensional feature vectors vertically or horizontally to perform regression classification.

[0046] Furthermore, the picture text information includes the size of the pixel value, the format of the picture and the coordinate values ​​corresponding to the entire picture content.

[0047] In a second aspect, the present invention provides a fast and accurate image detection and semantic classification system, comprising:

[0048] Image acquisition module: used to read images and image text information;

[0049] Image processing module: used to predict the coordinates of the upper left corner and the lower right corner of the image based on the image and the image data set information; select the range of the target image according to the coordinate values ​​of the upper left corner and the lower right corner of the image; read the Bbox value in the image text information, and analyze the coordinate values ​​of each image in the image; based on the coordinate values ​​of each image in the image, determine whether the target image is in the selected prediction rectangle within the range of the target image, wherein, when all features in the target image exceeding the specified proportion are located in the prediction rectangle, it is divided into a valid prediction rectangle;

[0050] Image analysis module: used to use the VGG19 neural network model to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size to obtain a multi-dimensional feature vector; regress the multi-dimensional feature vector to the corresponding image type through the softmax function in the neural network model, and obtain the predicted probability of belonging to each of the image types; construct the first loss function and the second loss function of the upper left corner point and the lower right corner point according to the Bbox value of the image text information in each of the image types;

[0051] Optimization module: used to optimize the recognition results of the effective prediction rectangle according to the first loss function and the second loss function in sequence, and then select the RMSprop optimizer to reduce the error value;

[0052] Training module: used to build training steps and test steps at the same time, set the number of training and optimization times, and iterate;

[0053] Result output module: used to compare the predicted probability obtained after iterative optimization with the set threshold, and output the predicted probability greater than the set threshold as the result.

[0054] In a third aspect, the present invention provides a fast and accurate image detection and semantic classification device, including a processor and a storage medium;

[0055] The storage medium is used to store instructions;

[0056] The processor is used to operate according to the instructions to execute the steps of any of the methods described above.

[0057] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] A fast and accurate image detection and semantic classification method provided by the present invention is improved and optimized on the basis of general Anchor-Free (a target detection network) target detection. Compared with the current methods for image detection and semantic classification, the Bbox value in the image text information can be directly read to accurately frame the position of the image in the image. The VGG19 neural network model is used to further compress the image size, thereby accelerating the speed of image detection and semantic classification. The image input is passed through two loss functions, and an optimizer is selected for repeated iterative optimization. The number of overall iterations can be increased or the learning rate can be reduced according to the error value and accuracy of the output result, so as to obtain the best image detection and semantic classification model, thereby improving the accuracy of the detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the image detection and semantic classification method provided by the first embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of the coordinates of the upper left corner and the lower right corner in the Bbox provided by the first embodiment of the present invention;

[0062] Figure 3 This is a flowchart of the VGG19 neural network of an input image provided by the first embodiment of the present invention;

[0063] Figure 4 is a training error model diagram provided in Embodiment 1 of the present invention;

[0064] Figure 5 It is a flow chart of the picture detection and semantic classification technical solution provided in the first embodiment of the present invention.

[0065] Figure 6 Schematic diagram of the topological structure of the target detection method provided in the first embodiment of the present invention; DETAILED DESCRIPTION

[0066] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0067] Embodiment 1:

[0068] The present embodiment provides a fast and accurate image detection and semantic classification method, including reading and loading images and various image data sets, and obtaining text information of the images. We can cluster the entire image collection by building a model, divide the clusters into smaller subclasses, design labels for these subclasses, and directly read the Bbox value in the image text information (which represents the coordinate value of each image in the image), accurately frame the specific position of the image in the image, and then perform convolution, maximum pooling, expansion and linearization steps through a neural network to extract the main features of the image, and train and optimize the model at the same time, thereby greatly improving the processing speed of the image. In this way, the entire chaotic image data set can be divided into smaller categories with clearer semantic information and more refined categories, thereby improving the accuracy of image detection and speeding up processing.

[0069] The specific process of this method in the algorithm includes:

[0070] Load the image data set and create two categories: image training data set and image test data set; obtain the length of the training data set and the test data set, and set the parameter batch_size (the number of images processed in batches) to 64; use the VGG19 neural network model, set the pixel size of the input image, the image size used in the present invention is 32*32 pixels, the number of input channels is 3, the number of output channels is 32, the convolution kernel size is 5, the step size is 1, and convolution layer 1, pooling layer 1, convolution layer 2, pooling layer 2, expansion and linearization processing are performed to extract feature points and compress the image size. The obtained image size is 10*10 pixels, which can reduce the use of storage space; select the loss function CrossEntropyLoss (cross entropy) and MSELoss (square difference), set the parameter learning The learning rate is 0.01, and the RMSprop (a method for deep learning gradient calculation) optimizer is called to process the input image; the number of training and testing are recorded respectively, and the number of training is set; the training step starts at this time, and when the default parameters of the network model need to be cleared, the test step starts; the multi-dimensional feature vector is regressed and classified into the corresponding image type through the softmax (normalized exponential function) function in the neural network model, and the image type with the largest probability value is determined as the current image type of the image; at this time, the accuracy on the overall test set can be obtained by the number of correct image types / the length of the test data set or the training data set; and the number of overall iterations can be increased or the learning rate can be reduced to improve the accuracy according to the error value and accuracy of the output result. After thousands of iterative optimizations of the model, the optimal image detection and semantic classification model can be obtained at this time. The ImageDraw (picture drawing) module is called in the above-mentioned trained optimal model to directly read the Bbox value in the image text information, and the retangle (matrix) module is used to accurately read the position of the image in the image according to the Bbox value. At this time, the speed and accuracy of image detection and semantic classification can be further accelerated.

[0071] See also Figure 1 , the specific steps to implement the above technical solution include:

[0072] S1. Obtain various images and image data set information, and predict the coordinates of the upper left corner and lower right corner of the image to be processed. In this embodiment, you can download some pre-trained, labeled large image data sets such as COCO, ImageNet, etc. in PyTorch-Torchvision-Datasets, or make your own image data set online in CVAT, label the images, classify them, and export the image data set. Then, use the VGG19 neural network model, set the training model and the test model, select the RMSprop optimizer to optimize the Loss, and then train the model hundreds or thousands of times to obtain a network model with a higher accuracy rate. This model can be used for image detection.

[0073] S2. According to the image coordinate values ​​of the upper left corner and the lower right corner, the range of the target image can be selected, and within this range, it is determined whether the target image is in the selected prediction rectangular frame. If all features of the target image are in the prediction rectangular frame, it is classified as the best prediction rectangular frame; if most features of the target image are in the prediction rectangular frame, it is classified as a valid prediction rectangular frame; conversely, if a small part of the features or all the features of the target image are not in the prediction rectangular frame, it is classified as an invalid prediction rectangular frame;

[0074] S3. Through the Bbox value in the image information, it represents the coordinate value of each image in the image. After analyzing the specific position of each image in the image, we can use draw_bounding_boxes (drawing Bbox value boundary module) to draw a box on the image. The format of the box is (X min , Y min , X max , Y max ). The initial Bbox values ​​of the X-axis and Y-axis are the upper left corner of the image, and the initial Bbox values ​​of the X-axis and Y-axis are added with the height and width information to get the lower right corner of the image;

[0075] For example, the Bbox information in the coco annotation is given as (x, y, w, h), which represents the x-axis coordinate, y-axis coordinate, width and height respectively. In the algorithm, it needs to be converted into (x1, x2, y1, y2).

[0076] Set the coordinates of the upper left corner of an image to (0, 0) so that the conversion format is not prone to problems. At the same time, if the image is flipped horizontally, the coordinates will change, because (x1, y1) always represents the upper left corner of the Bbox. So when converting, check which is the new upper left corner and which is the new origin after the conversion, and then calculate the coordinates in the new coordinate system. Figure 2 It represents the schematic diagram of the coordinates of the upper left corner and the lower right corner in the Bbox.

[0077] S4. The VGG19 neural network model is used to calculate the multi-dimensional feature vector. Figure 3 , convolve the image in the neural network, perform maximum pooling, and then expand and linearize the image to compress the image capacity to the greatest extent and extract the main feature points, which can greatly speed up the recognition of the same type of images;

[0078] Exemplarily, the Conv2d module in the VGG19 neural network model refers to applying a two-dimensional convolution on an input signal composed of multiple input planes.

[0079]

[0080] Among them, * represents the 2D cross-correlation operator, out represents the output value, and input represents the input value. Indicates the number of output channels, bias indicates the bias value, weight indicates the weight, N i represents the number of images processed in the minimum batch size of the i-th image, C m Indicates the number of channels of the mth image, k indicates the convolution kernel size, and must be a positive integer;

[0081] The MaxPool2d module in the VGG19 neural network model refers to applying a 2D max pooling on an input signal consisting of several input planes.

[0082]

[0083] input(N i , C i , stride[0]×h+m, stride[1]×w+n)

[0084] N i represents the number of images processed in the minimum batch size of the i-th image, C i Indicates the number of channels of the i-th image, h indicates the height of the input plane in pixels; w indicates the width of the input plane in pixels; m and n indicate the number of images, kH is the height of the pooling kernel, and kW is the width of the pooling kernel. If the padding is non-zero, the input is implicitly padded with negative infinity on both sides, and stride indicates the step size of each step of the image during convolution;

[0085] The Flatten module in the VGG19 neural network model refers to flattening the continuous grayscale range into a tensor;

[0086] The Linear module in the VGG19 neural network model refers to applying linear transformation to the input data. This model only supports TensorFloat32;

[0087] y=XA T +B

[0088] Y represents the output value, X represents the input value, A is the input matrix, T represents the transpose of the matrix, and B represents the deviation value.

[0089] S5. Regressing the multi-dimensional feature vector into corresponding picture types through the softmax function in the neural network model, and obtaining the predicted probability of belonging to each picture type;

[0090] S6. Construct the first loss function of the upper left corner point and the lower right corner point according to the Bbox value of the image, and at the same time, construct the second loss function after one round;

[0091] The first loss function in S6 is CrossEntropyLoss, and the input should contain the raw, unnormalized scores for each class. For unbatched input, the input must be a tensor of size (C).

[0092] R(x,y)=L={R1,.R n .., R N} T

[0093] {yn≠ignore index value}

[0094] In the formula, x is the input value, y is the target value, L represents the entire matrix set, C is the number of classes, R n represents the matrix when the number is n, N is the number of images to be processed in the minimum batch, T represents the transposition of the matrix, yn represents the nth target value, and w yn represents the weight of the nth target value, x n,yn The input value when the number is n and the weight of the nth target value; x n,c Indicates the input value when the number is n and the image type is C; if the value of ignoring the index is given, the loss function will also follow the index value of this class. In this case, the unreduced loss value can be described as:

[0095]

[0096] The above formula takes the average value, and the following formula takes the sum.

[0097] Probability of each class: Applicable when more than one class label is required for each mini-batch, such as mixed labels, label smoothing, etc. The unreduced loss in this case can be described as:

[0098] R(x,y)=L={R1,.R n .., R N}T

[0099]

[0100] Where x is the input value, y is the target value, C is the number of classes, and w c represents the weight when the number of classes is C, N is the minimum size of the batch processed image, y n,c Represents the target value when the number of classes is C.

[0101] The second loss function in S6 is MSELoss, which is the mean square error between the input value X and each element of the target y.

[0102] R(x,y)=L={R1,.R n .., R N} T

[0103] R n =(x n -y n ) 2

[0104] x n Indicates the input value when the quantity is n, y n Represents the output value when the number is n, and L represents the entire set of matrices.

[0105] S7. Optimize the recognition result of the effective prediction frame according to the first loss function, and then import the optimization result into the second loss function to perform a second round of optimization on the recognition result of the above effective prediction frame; at the same time, construct training steps and test steps, set the number of training times and optimization times, and iterate, and determine the image type with the largest probability value as the current image type of the image, thereby improving the accuracy of the output result. In one embodiment, a softmax function can be used to calculate the multi-dimensional feature vector obtained by the neural network model, and the similarity of the two images can be determined by comparing them vertically or horizontally, and the image type with the largest probability value is determined as the current image type of the image.

[0106] S8. Compare the output value obtained after optimization with the set threshold, and output the output value greater than the set threshold as the result. For example, a similarity threshold (≧0.6) can be set at this time. When the matching degree between the upper left corner point and the lower right corner point constituting the valid prediction rectangle is greater than the threshold, the valid prediction rectangle is retained, otherwise it is discarded and the valid value is used as the output value. Figure 4This is the optimized model obtained after 78,000 iterations. The error of this model is only 0.02, which has very high precision and can greatly improve the accuracy of target detection and image classification. It only takes 13ms to process a picture, and the accuracy rate is close to 70%, achieving the purpose of fast and accurate.

[0107] Figure 5 The figure is a flowchart of the picture detection and semantic classification technical solution in an embodiment of the present invention.

[0108] After importing the prepared image dataset, the Bbox value in the image text information can be directly read by calling the ImageDraw module, and the position of the image in the image can be accurately read by the retangle module according to the Bbox value to construct a predicted rectangular frame; create two categories of image training dataset and image test dataset; obtain the length of the training dataset and the test dataset; use the VGG19 neural network model to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size; select the loss function, adjust the learning rate, and use the RMSprop optimizer to process the input image; record the number of training and testing respectively, and set the number of training; the training step starts at this time, and the test step starts when the default parameters of the network model need to be cleared; the multi-dimensional feature vector is regressed and classified into the corresponding image type through the softmax function in the neural network model, and the image type with the largest probability value is determined as the current image type of the image; at this time, the accuracy on the overall test set can be obtained by the correct number of image types / the length of the test dataset or the training dataset, and the output value obtained after optimization is compared with the set threshold, and the output value greater than the set threshold is output as the result.

[0109] Embodiment 2:

[0110] See also Figure 6 A fast and accurate image detection and semantic classification system can implement a fast and accurate image detection and semantic classification method described in Example 1, and mainly includes six modules: an image acquisition module, an image processing module, an image analysis module, an optimization module, a training module and a result output module, wherein:

[0111] Image acquisition module: used to read images and image text information;

[0112] Image processing module: used to predict the coordinates of the upper left corner and the lower right corner of the image based on the image and the image data set information; select the range of the target image according to the coordinate values ​​of the upper left corner and the lower right corner of the image; read the Bbox value in the image text information, and analyze the coordinate values ​​of each image in the image; based on the coordinate values ​​of each image in the image, determine whether the target image is in the selected prediction rectangle within the range of the target image, wherein, when all features in the target image exceeding the specified proportion are located in the prediction rectangle, it is divided into a valid prediction rectangle;

[0113] Image analysis module: used to use the VGG19 neural network model to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size to obtain a multi-dimensional feature vector; regress the multi-dimensional feature vector to the corresponding image type through the softmax function in the neural network model, and obtain the predicted probability of belonging to each of the image types; construct the first loss function and the second loss function of the upper left corner point and the lower right corner point according to the Bbox value of the image text information in each of the image types;

[0114] Optimization module: used to optimize the recognition results of the effective prediction rectangle according to the first loss function and the second loss function in sequence, and then select the RMSprop optimizer to reduce the error value;

[0115] Training module: used to build training steps and test steps at the same time, set the number of training and optimization times, and iterate;

[0116] Result output module: used to compare the predicted probability obtained after iterative optimization with the set threshold, and output the predicted probability greater than the set threshold as the result.

[0117] To build this model, you can prepare data, load data, build a network model, set the loss function, set the optimizer, start training, start testing, output the accuracy, display the results, and save the model in an integrated development environment.

[0118] Embodiment three:

[0119] The embodiment of the present invention also provides a fast and accurate image detection and semantic classification device, which can implement the fast and accurate image detection and semantic classification method described in the first embodiment, including a processor and a storage medium;

[0120] The storage medium is used to store instructions;

[0121] The processor is used to operate according to the instructions to perform the steps of the following method:

[0122] Read the image and its text information, and predict the coordinates of the upper left corner and lower right corner of the image;

[0123] According to the coordinate values ​​of the upper left corner and the lower right corner of the image, select the range of the target image;

[0124] Read the Bbox value in the image text information and analyze the coordinate values ​​of each image in the image;

[0125] Based on the coordinate values ​​of each image in the image, determine whether the target image is in the selected prediction rectangular frame within the range of the target image, wherein when all features exceeding a specified proportion in the target image are located in the prediction rectangular frame, it is divided into a valid prediction rectangular frame;

[0126] The VGG19 neural network model is used to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size to obtain a multi-dimensional feature vector;

[0127] Regressing the multi-dimensional feature vector into corresponding picture types through the softmax function in the neural network model, and obtaining the predicted probability of belonging to each picture type;

[0128] Constructing a first loss function and a second loss function for the upper left corner point and the lower right corner point according to the Bbox value of the picture text information in each of the picture types;

[0129] The recognition result of the effective prediction rectangle is optimized according to the first loss function and the second loss function in sequence, and then the RMSprop optimizer is selected to reduce the error value;

[0130] Construct training steps and test steps at the same time, set the number of training and optimization times, and iterate;

[0131] The predicted probability obtained after iterative optimization is compared with the set threshold, and the predicted probability greater than the set threshold is output as the result.

[0132] Embodiment 4:

[0133] The embodiment of the present invention further provides a computer-readable storage medium, which can implement the fast and accurate image detection and semantic classification method described in the first embodiment, and stores a computer program thereon, which implements the steps of the following method when executed by a processor:

[0134] Read the image and its text information, and predict the coordinates of the upper left corner and lower right corner of the image;

[0135] According to the coordinate values ​​of the upper left corner and the lower right corner of the image, select the range of the target image;

[0136] Read the Bbox value in the image text information and analyze the coordinate values ​​of each image in the image;

[0137] Based on the coordinate values ​​of each image in the image, determine whether the target image is in the selected prediction rectangular frame within the range of the target image, wherein when all features exceeding a specified proportion in the target image are located in the prediction rectangular frame, it is divided into a valid prediction rectangular frame;

[0138] The VGG19 neural network model is used to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size to obtain a multi-dimensional feature vector;

[0139] Regressing the multi-dimensional feature vector into corresponding picture types through the softmax function in the neural network model, and obtaining the predicted probability of belonging to each picture type;

[0140] Constructing a first loss function and a second loss function for the upper left corner point and the lower right corner point according to the Bbox value of the picture text information in each of the picture types;

[0141] The recognition result of the effective prediction rectangle is optimized according to the first loss function and the second loss function in sequence, and then the RMSprop optimizer is selected to reduce the error value;

[0142] Construct training steps and test steps at the same time, set the number of training and optimization times, and iterate;

[0143] The predicted probability obtained after iterative optimization is compared with the set threshold, and the predicted probability greater than the set threshold is output as the result.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0145] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fast and accurate image detection and semantic classification method, characterized by: include: Read the image and its text information, and predict the coordinates of the upper left corner and lower right corner of the image; According to the coordinate values ​​of the upper left corner and the lower right corner of the image, select the range of the target image; Read the Bbox value in the image text information and analyze the coordinate values ​​of each image in the image; Based on the coordinate values ​​of each image in the image, determine whether the target image is in the selected prediction rectangular frame within the range of the target image, wherein when all features exceeding a specified proportion in the target image are located in the prediction rectangular frame, it is divided into a valid prediction rectangular frame; The VGG19 neural network model is used to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size to obtain a multi-dimensional feature vector; Regressing the multi-dimensional feature vector into corresponding picture types through the softmax function in the neural network model, and obtaining the predicted probability of belonging to each picture type; Constructing a first loss function and a second loss function for the upper left corner point and the lower right corner point according to the Bbox value of the picture text information in each of the picture types; The recognition result of the effective prediction rectangle is optimized according to the first loss function and the second loss function in sequence, and then the RMSprop optimizer is selected to reduce the error value; Construct training steps and test steps at the same time, set the number of training and optimization times, and iterate; The predicted probability obtained after iterative optimization is compared with the set threshold, and the predicted probability greater than the set threshold is output as the result.

2. The fast and accurate image detection and semantic classification method according to claim 1, characterized in that: If all features of the target image are located in the prediction rectangular frame, it is classified as the best prediction rectangular frame; if features of the target image exceeding the specified proportion are not located in the prediction rectangular frame or all features are not in the prediction rectangular frame, it is classified as an invalid prediction rectangular frame.

3. The fast and accurate image detection and semantic classification method according to claim 1, characterized in that: The predicted rectangular frame is constructed by calling the ImageDraw module to directly read the Bbox value in the picture text information, and using the retangle module to accurately read the position of the image in the picture according to the Bbox value.

4. The fast and accurate image detection and semantic classification method according to claim 1, characterized in that: The VGG19 neural network model: Convolution is performed through the Conv2d module, which applies a two-dimensional convolution on the input signal composed of multiple input planes. The expression is: Among them, * represents the 2D cross-correlation operator, out represents the output value, and input represents the input value. Indicates the number of output channels, bias indicates the bias value, weight indicates the weight, N i represents the number of images processed in the minimum batch size of the i-th image, C m Indicates the number of channels of the mth image, k indicates the convolution kernel size, and must be a positive integer; Max pooling is performed through the MaxPool2d module, which applies a 2D max pool on the input signal composed of several input planes. The expression is: input(N i ,C i ,stride[0]×h+m,stride[1]×w+n) N i represents the number of images processed in the minimum batch size of the i-th image, C i Indicates the number of channels of the i-th image, h indicates the height of the input plane in pixels; w indicates the width of the input plane in pixels; m and n indicate the number of images, kH is the height of the pooling kernel, kW is the width of the pooling kernel, if the padding is non-zero, the input is implicitly padded with negative infinity on both sides, and stride indicates the step size of each step of the image during convolution; The continuous grayscale range is flattened into a tensor through the Flatten module; Linearization is performed through the Linear module to apply a linear transformation to the input data. The expression is: Y=XA T +B Y represents the output value, X represents the input value, A is the input matrix, T represents the transpose of the matrix, and B represents the deviation value.

5. The fast and accurate image detection and semantic classification method according to claim 1, characterized in that: The first loss function is CrossEntropyLoss, and its expression is: R(x,y)=L={R1,.R n ..,R N } T {yn≠ignore index value} In the formula, x is the input value, y is the target value, L represents the entire matrix set, C is the number of classes, R n represents the matrix when the number is n, N is the number of images to be processed in the minimum batch, T represents the transposition of the matrix, yn represents the nth target value, and w yn represents the weight of the nth target value, x n,yn The input value when the number is n and the weight of the nth target value; x n,c Indicates the input value when the number is n and the type of image is C; When given a value for the ignore index, the unreduced loss is described as: The above formula takes the average value, and the following formula takes the sum; Probability of each class: Applicable when more than one class label is required for each mini-batch, the unreduced loss is described as: R(x,y)=L={R1,.R n ..,R N } T In the formula, x is the input value, y is the target value, C is the number of classes, and w c represents the weight when the number of classes is C, N is the minimum size of the batch processed image, y n,c Indicates the target value when the number of classes is C; The second loss function is MSELoss, and the expression is: R(x,y)=L={R1,.R n ..,R N } T R n =(x n -y n ) 2 Among them, x n Indicates the input value when the quantity is n, y n Represents the output value when the number is n, and L represents the entire set of matrices.

6. The fast and accurate image detection and semantic classification method according to claim 1, characterized in that: The softmax function determines the similarity between two images by comparing the multi-dimensional feature vectors vertically or horizontally to perform regression classification.

7. The fast and accurate image detection and semantic classification method according to claim 1, characterized in that: The image text information includes the size of the pixel value, the format of the image and the coordinate values ​​corresponding to the entire image content.

8. A fast and accurate image detection and semantic classification system, characterized by: include: Image acquisition module: used to read images and image text information; Image processing module: used to predict the coordinates of the upper left corner and the lower right corner of the image based on the image and the image data set information; select the range of the target image according to the coordinate values ​​of the upper left corner and the lower right corner of the image; Read the Bbox value in the image text information and analyze the coordinate values ​​of each image in the image; Based on the coordinate values ​​of each image in the image, determine whether the target image is in the selected prediction rectangular frame within the range of the target image, wherein when all features exceeding a specified proportion in the target image are located in the prediction rectangular frame, it is divided into a valid prediction rectangular frame; Image analysis module: used to use the VGG19 neural network model to perform convolution, maximum pooling, expansion and linearization on the image, extract feature points and compress the image size to obtain a multi-dimensional feature vector; regress the multi-dimensional feature vector to the corresponding image type through the softmax function in the neural network model, and obtain the predicted probability of belonging to each of the image types; construct the first loss function and the second loss function of the upper left corner point and the lower right corner point according to the Bbox value of the image text information in each of the image types; Optimization module: used to optimize the recognition results of the effective prediction rectangle according to the first loss function and the second loss function in sequence, and then select the RMSprop optimizer to reduce the error value; Training module: used to build training steps and test steps at the same time, set the number of training and optimization times, and iterate; Result output module: used to compare the predicted probability obtained after iterative optimization with the set threshold, and output the predicted probability greater than the set threshold as the result.

9. A fast and accurate image detection and semantic classification device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images