Target classification method and device based on image analysis and electronic equipment

Abstract

The invention discloses a target classification method and device based on image analysis and electronic equipment, and relates to the field of image recognition. The method comprises the following steps: acquiring a target image containing a to-be-identified bottle; performing target detection on the target image to obtain a target bottle, and extracting a boundary contour of the target bottle; determining the angle and gradient of the target bottle based on the boundary contour; according to the angle and the gradient, correcting the target bottle to reach a preset standard angle and a preset standard gradient, and obtaining a corrected standard image; based on the target bottle, carrying out characteristic extraction on the standard image to obtain a target characteristic; and if it is determined that the similarity between the target feature and the target bottom library feature is greater than or equal to a preset similarity threshold, determining a target bottle category corresponding to the target bottom library feature, and taking the target bottle category as a bottle category corresponding to the to-be-identified bottle. By implementing the technical scheme provided by the invention, the accuracy of bottle classification is improved by performing standardization processing on the target image.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and specifically to a target classification method, device and electronic equipment based on image analysis. Background Art

[0002] With the rapid development of automation technology, robotic arms have been widely used in various industrial fields, especially in the task of sorting and handling items. In the production line, robotic arms can effectively sort various bottles, improve production efficiency and reduce the need for manual operation. The application of this technology, especially in the food and beverage industry, is of great significance for maintaining product quality and optimizing production processes.

[0003] At present, the bottle classification method in the related technology mainly relies on the visual system, which faces challenges in complex practical application environments. The related visual system often performs poorly when dealing with situations where the bottle posture changes, because the angle and inclination of the bottle will affect the accuracy of image recognition. Therefore, the visual detection method in the related technology has the problem of low accuracy.

[0004] Therefore, there is an urgent need for a target classification method, device and electronic equipment based on image analysis. Summary of the invention

[0005] The present application provides a target classification method, device and electronic equipment based on image analysis, which improves the accuracy of bottle classification by standardizing the target image.

[0006] In a first aspect of the present application, a target classification method based on image analysis is provided, the method comprising: acquiring a target image containing a bottle to be identified; performing target detection on the target image to obtain a target bottle, and extracting a boundary contour of the target bottle; determining the angle and inclination of the target bottle based on the boundary contour; correcting the target bottle according to the angle and the inclination to achieve a preset standard angle and a preset standard inclination, and obtaining a corrected standard image; performing feature extraction on the standard image based on the target bottle to obtain a target feature; comparing the target feature with a preset feature base library, if it is determined that the similarity between the target feature and the target base library feature is greater than or equal to a preset similarity threshold, determining a target bottle category corresponding to the target base library feature, and using the target bottle category as the bottle category corresponding to the bottle to be identified, the preset feature base library includes multiple base library features, and the target base library feature is any one of the multiple base library features.

[0007] By adopting the above technical solution, by acquiring a target image containing a bottle to be identified, performing target detection and boundary contour extraction on the target image, the position and range of the target bottle can be accurately located. Then, the angle and inclination of the target bottle are determined based on the boundary contour, and correction is performed so that the target bottle reaches a preset standard angle and a preset standard inclination. This step can eliminate the image differences caused by different shooting angles and bottle placement postures, and improve the accuracy of subsequent feature extraction and comparison. Next, feature extraction is performed on the corrected standard image to obtain target features that can characterize the characteristics of the target bottle. Finally, the target features are compared with the preset feature base library to find the target base library features with the highest similarity, and the corresponding bottle category is used as the recognition result. The technical solution obtains a standard image by correcting the target image, ensures that the target bottle has a standard posture before feature extraction, improves the accuracy of feature matching, and effectively classifies the bottle to be identified in the target image and identifies the bottle category to which it belongs.

[0008] Optionally, determining the angle and inclination of the target bottle based on the boundary contour specifically includes: performing key point detection on the target bottle to obtain key point coordinates of the target bottle; determining a first key point and a second key point from the key point coordinates, wherein the first key point is located at the top of the target bottle and the second key point is located at the bottom of the target bottle; calculating a first angle between a line connecting the first key point and the second key point and a horizontal direction, and determining the first angle as the angle of the target bottle; calculating a second angle between a line connecting the first key point and the second key point and a vertical direction, and determining the second angle as the inclination of the target bottle.

[0009] By adopting the above technical solution, by detecting the key points of the target bottle, the key point coordinates that can characterize the shape and posture of the bottle can be obtained. Among them, the two key points (the first key point and the second key point) located at the top and bottom of the bottle can reflect the overall direction and tilt of the bottle. By calculating the angle between the line connecting the two key points and the horizontal direction, the rotation angle of the target bottle in the target image can be obtained; by calculating the angle between the line connecting the two key points and the vertical direction, the tilt degree of the target bottle can be obtained. This method of key point detection and angle calculation can accurately quantify the posture information of the bottle.

[0010] Optionally, the target bottle is corrected according to the angle and the inclination to reach a preset standard angle and a preset standard inclination, and a corrected standard image is obtained, specifically comprising: comparing the angle of the target bottle with a preset standard angle to obtain an angle difference, and comparing the inclination of the target bottle with a preset standard inclination to obtain an inclination difference; constructing an affine transformation matrix according to the angle difference and the inclination difference; and performing an affine transformation on the target bottle using the affine transformation matrix to make the angle of the target bottle reach the preset standard angle, and the inclination of the target bottle reach the preset standard inclination, and a corrected standard image is obtained.

[0011] By adopting the above technical solution, the angle of the target bottle is compared with the preset standard angle to obtain the angle difference, and the inclination of the target bottle is compared with the preset standard inclination to obtain the inclination difference, thereby quantifying the degree to which the bottle posture deviates from the standard state. Then, an affine transformation matrix is ​​constructed according to the angle difference and the inclination difference, and the target image is affine transformed using the affine transformation matrix so that its angle reaches the preset standard angle and its inclination reaches the preset standard inclination. Through the affine transformation, the target images in different postures can be uniformly converted into target images in standard postures, eliminating the influence of posture differences on subsequent processing. The standardized bottle image has a uniform angle and inclination, which is convenient for feature extraction and comparison.

[0012] Optionally, before acquiring the target image containing the bottle to be identified, the method further includes: acquiring a plurality of initial images; performing data amplification on the initial images to obtain amplified images, wherein the data amplification includes at least one of rotation, scaling, cropping, flipping and color transformation; using the amplified images as training samples, and training the amplified images using a preset convolutional model to obtain the preset feature base library.

[0013] By adopting the above technical solution, a large number of augmented images can be obtained by obtaining the initial image and performing data augmentation operations such as rotation, scaling, cropping, flipping and color transformation on it. Data augmentation can effectively increase the number and diversity of training samples. The augmented images cover a wider range of bottle shapes, angles, and sizes. Using the augmented images as training samples and training with the preset convolutional model, an optimized feature extractor can be obtained. After training with a large number of augmented images, the preset convolutional model can learn a generalized bottle feature representation and form a preset feature base library. The preset feature base library contains typical features of different types of bottles and can serve as the basis for subsequent recognition and classification.

[0014] Optionally, the amplified image is used as a training sample, and a preset convolution model is used to train the amplified image to obtain the preset feature base library, which specifically includes: using the preset convolution model to extract features of the amplified image to obtain multiple training features; performing cluster analysis on the multiple training features to obtain multiple cluster centers, and using the multiple cluster centers as base library features in the preset feature base library.

[0015] By adopting the above technical solution, the feature vector representation of each amplified image can be obtained by extracting features from the amplified image through the preset convolution model. Cluster analysis is performed on the feature vectors extracted from multiple training images to discover the inherent structure and pattern in the feature space. The cluster center is used as the base library feature in the preset feature base library. The preset feature base library covers the common features of different types of bottles and can effectively represent and compare unknown bottle images.

[0016] Optionally, performing target detection on the target image to obtain a target bottle and extracting a boundary contour of the target bottle specifically includes: performing target detection on the target image using a preset target detection algorithm to obtain position information of the target bottle; extracting an image region of the target bottle from the target image based on the position information; and performing instance segmentation on the image region of the target bottle using a preset instance segmentation model to obtain a boundary contour of the target bottle.

[0017] By adopting the above technical solution, the target image is scanned and identified by using a preset target detection algorithm, and the position information of the target bottle in the target image, including the coordinates and size of the bounding box, can be obtained. The preset target detection algorithm can automatically locate and identify the target bottle from the target image by learning a large amount of annotated data, and can maintain a high detection accuracy even in complex backgrounds and multi-target situations. According to the detected position information, the image area of ​​the corresponding target bottle can be extracted from the original image, excluding the influence of the background and other interfering objects. Instance segmentation of the extracted target bottle image area can obtain a more refined and accurate boundary contour of the target bottle.

[0018] Optionally, the target bottle category includes a recyclable bottle category and a non-recyclable bottle category. After taking the target bottle category as the bottle category corresponding to the bottle to be identified, the method further includes: if the target bottle category is a recyclable bottle category, importing the bottle to be identified into a recycling bin; if the target bottle category is a non-recyclable bottle category, importing the bottle to be identified into a garbage bin.

[0019] By adopting the above technical solution, by classifying and identifying the target bottles, it can be determined whether they are recyclable bottles or non-recyclable bottles. This judgment result is of great significance for the subsequent bottle processing and resource utilization. For recyclable bottles, they are automatically imported into the recycling station for centralized recycling and reuse. For non-recyclable bottles, they are imported into the garbage station for centralized processing. The automated processing solution based on the target bottle category realizes the intelligent and standardized bottle recycling and disposal, and improves the resource utilization efficiency and environmental protection level.

[0020] In a second aspect of the present application, a target classification device based on image analysis is provided, the device comprising an acquisition module and a processing module, the acquisition module is used to acquire a target image containing a bottle to be identified; the processing module is used to perform target detection on the target image to obtain a target bottle and extract a boundary contour of the target bottle; the processing module is also used to determine the angle and inclination of the target bottle based on the boundary contour; the processing module is also used to correct the target bottle according to the angle and the inclination to achieve a preset standard angle and a preset standard inclination, and obtain a corrected standard image; the processing module is also used to extract features from the standard image based on the target bottle to obtain a target feature; the processing module is also used to compare the target feature with a preset feature base library, if it is determined that the similarity between the target feature and the target base library feature is greater than or equal to a preset similarity threshold, then determine the target bottle category corresponding to the target base library feature, and use the target bottle category as the bottle category corresponding to the bottle to be identified, the preset feature base library includes multiple base library features, and the target base library feature is any one of the multiple base library features.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By acquiring a target image containing the bottle to be identified, performing target detection and boundary contour extraction on the target image, the position and range of the target bottle can be accurately located. Then, the angle and inclination of the target bottle are determined based on the boundary contour, and correction is performed so that the target bottle reaches the preset standard angle and preset standard inclination. This step can eliminate the image differences caused by different shooting angles and bottle placement postures, and improve the accuracy of subsequent feature extraction and comparison. Next, feature extraction is performed on the corrected standard image to obtain target features that can characterize the characteristics of the target bottle. Finally, the target features are compared with the preset feature base library to find the target base library features with the highest similarity, and the corresponding bottle category is used as the recognition result. This technical solution obtains a standard image by correcting the target image, ensures that the target bottle has a standard posture before feature extraction, improves the accuracy of feature matching, and effectively classifies the bottles to be identified in the target image and identifies the bottle category to which it belongs.

[0024] 2. By performing key point detection on the target bottle, the key point coordinates that can characterize the shape and posture of the bottle can be obtained. Among them, the two key points (the first key point and the second key point) located at the top and bottom of the bottle can reflect the overall direction and tilt of the bottle. By calculating the angle between the line connecting the two key points and the horizontal direction, the rotation angle of the target bottle in the target image can be obtained; by calculating the angle between the line connecting the two key points and the vertical direction, the tilt degree of the target bottle can be obtained. This method of key point detection and angle calculation can accurately quantify the posture information of the bottle.

[0025] 3. By comparing the angle of the target bottle with the preset standard angle, the angle difference is obtained. By comparing the inclination of the target bottle with the preset standard inclination, the inclination difference is obtained, thereby quantifying the degree to which the bottle posture deviates from the standard state. Then, an affine transformation matrix is ​​constructed based on the angle difference and the inclination difference, and the affine transformation matrix is ​​used to perform an affine transformation on the target image so that its angle reaches the preset standard angle and its inclination reaches the preset standard inclination. Through affine transformation, target images in different postures can be uniformly converted into target images in standard postures, eliminating the influence of posture differences on subsequent processing. The standardized bottle image has a uniform angle and inclination, which is convenient for feature extraction and comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a target classification method based on image analysis disclosed in an embodiment of the present application; Figure 2 It is a module schematic diagram of a target classification device based on image analysis disclosed in an embodiment of the present application; Figure 3It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0027] Explanation of the reference numerals: 201, acquisition module; 202, processing module; 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0028] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0029] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0030] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0031] This application provides a target classification method based on image analysis, referring to Figure 1 , Figure 1 1 is a flow chart of a method for classifying a target based on image analysis provided in an embodiment of the present application. The method is applied to a controller of a robotic arm, and the method includes steps S101 to S106, which are as follows: Step S101: Acquire a target image containing a bottle to be identified.

[0032] In step S101, a recycling station uses an automatic robot arm to classify items to identify the bottles to be identified. The robot arm has a built-in target analysis program based on image analysis, which is executed by the controller of the robot arm. When the bottle to be identified appears on the conveyor belt of the recycling station, the controller will receive a signal from the sensor and know that the bottle to be identified has entered the working range of the robot arm. At this time, the controller will start the camera, trigger the camera to take a picture, and obtain a target image containing the bottle to be identified.

[0033] In a possible implementation, before step S101, the method further includes: acquiring a plurality of initial images; performing data amplification on the initial images to obtain amplified images, wherein the data amplification includes at least one of rotation, scaling, cropping, flipping, and color transformation; using the amplified images as training samples, and training the amplified images using a preset convolutional model to obtain a preset feature base library.

[0034] Specifically, the controller obtains multiple initial images from the image database, and then applies various data augmentation techniques to these initial images, such as rotation, scaling, cropping, flipping, and color transformation, to generate augmented images. Through data augmentation, the angle, size, position, and lighting conditions of the bottle in the actual scene can be simulated. The controller uses the augmented images as training samples and uses the preset convolution model for training to obtain the preset feature base library for feature extraction and comparison.

[0035] In a possible implementation, the amplified image is used as a training sample, and a preset convolution model is used to train the amplified image to obtain a preset feature base library, specifically including: using the preset convolution model to extract features of the amplified image to obtain multiple training features; performing cluster analysis on the multiple training features to obtain multiple cluster centers, and using the multiple cluster centers as base library features in the preset feature base library.

[0036] After obtaining the augmented image, the controller inputs it into the preset convolution model for feature extraction. During the feature extraction process, the convolution network converts the augmented image into a more abstract feature representation layer by layer, and outputs a high-dimensional feature vector at the last layer of the network. The controller extracts the feature vector for each augmented image to obtain a training feature set.

[0037] In order to mine representative feature patterns from the training feature set, the controller will perform cluster analysis on all feature vectors. Cluster analysis is an unsupervised learning method that aims to classify similar features into one category and dissimilar features into different categories. The controller uses the K-means clustering algorithm for clustering. During the clustering process, the K-means clustering algorithm automatically searches for dense areas in the feature space and uses the center points of the dense areas as cluster centers, which are typical representatives of each feature category. Through cluster analysis, the controller can obtain several cluster centers, each of which corresponds to a characteristic pattern of a bottle. The controller will use these cluster centers as the base library features in the preset feature base library and assign a unique identification number to each cluster center.

[0038] For example, the controller can obtain 1,000 initial images of various bottles from the image database, and then perform 10 different data augmentation operations on each image, such as rotation ±30°, scaling 0.8-1.2 times, cropping 10-20%, horizontal / vertical flipping, brightness / contrast / saturation adjustment, etc., and finally obtain 10,000 augmented images. The controller uses these augmented images as training samples and uses the preset ResNet-50 convolutional neural network model for training to obtain a preset feature base library, which includes base library features of various bottle categories.

[0039] Step S102: performing target detection on the target image to obtain a target bottle and extracting a boundary contour of the target bottle.

[0040] In step S102, a preset target detection algorithm is used to perform target detection on the target image to obtain the position information of the target bottle; based on the position information, an image area of ​​the target bottle is extracted from the target image; and a preset instance segmentation model is used to perform instance segmentation on the image area of ​​the target bottle to obtain a boundary contour of the target bottle.

[0041] Specifically, the controller uses a preset target detection algorithm to analyze the target image. The controller loads a pre-trained target detection model, such as YOLO, and inputs the target image into the target detection model for forward reasoning. The target detection model extracts multi-scale features of the target image through a convolutional neural network, and then gradually reduces the size of the feature map through a series of convolutional layers and pooling layers, and finally obtains a set of candidate regions and their corresponding category probabilities. The controller selects the candidate regions with higher scores according to a preset threshold (such as 0.5) as the location information of the detected target image, which is usually expressed as the coordinates of a rectangular bounding box (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the bounding box, w is the width of the bounding box, and h is the height of the bounding box.

[0042] After obtaining the position information of the target image, the controller extracts the corresponding image area from the original target image according to the position information. Specifically, the controller crops a sub-image with a width of w and a height of h from the target image according to the coordinates (x, y, w, h) of the bounding box, and the coordinates of the upper left corner of the sub-image are (x, y). In this way, the cropped sub-image only contains the image area of ​​the target bottle (target bottle), excluding most of the irrelevant background information.

[0043] The controller then uses a preset instance segmentation model to further segment the cropped image area of ​​the target bottle. The controller uses a preset instance segmentation model, such as MaskR-CNN, and inputs the image area of ​​the target bottle into the preset instance segmentation model for forward reasoning. The preset instance segmentation model extracts the features of the image area through a convolutional neural network, and then generates the category, bounding box and binary mask of the target image through one or more fully connected layers. Among them, the binary mask is the same size as the image area, the pixel value belonging to the target image is 1, and the background pixel value is 0, thereby obtaining the boundary outline of the target image. The controller performs a dot multiplication of the binary mask and the original image area to obtain a segmentation result image that only contains the target bottle, that is, the boundary outline of the target bottle.

[0044] For example, the controller receives a 1000×800 pixel target image, which contains a glass beer bottle located in the center of the image. The controller first uses the YOLOv3 model to detect the target image and obtains the bounding box coordinates of the beer bottle as (300, 200, 400, 500), that is, the upper left corner coordinates are (300, 200), the width is 400 pixels, and the height is 500 pixels. The controller cuts out a 400×500 beer bottle image from the original image based on the bounding box. Then, the controller uses the pre-trained MaskR-CNN model to perform instance segmentation on the sub-image and generates a 400×500 binary mask, where the pixel value belonging to the beer bottle is 1 and the background pixel value is 0. The controller performs a dot multiplication of the binary mask and the sub-image to obtain a segmentation result image containing only the beer bottle area, with the pixel value being the original RGB value and the background pixel value being 0. Finally, the controller performs contour extraction on the segmentation result image to obtain a closed polygon curve, which depicts the boundary contour of the beer bottle.

[0045] Step S103: Determine the angle and inclination of the target bottle based on the boundary contour.

[0046] In step S103, key point detection is performed on the target bottle to obtain the key point coordinates of the target bottle; the first key point and the second key point are determined from the key point coordinates, the first key point is located at the top of the target bottle, and the second key point is located at the bottom of the target bottle; a first angle between a line connecting the first key point and the second key point and the horizontal direction is calculated, and the first angle is determined as the angle of the target bottle; a second angle between a line connecting the first key point and the second key point and the vertical direction is calculated, and the second angle is determined as the inclination of the target bottle.

[0047] Specifically, the controller performs key point detection on the boundary contour of the target image. Key points refer to points on an object with obvious geometric or topological features, such as vertices, base points, and center points. The controller uses the SIFT (Scale Invariant Feature Transform) key point detection algorithm to analyze the boundary contour. The SIFT algorithm calculates the gradient, curvature, entropy and other features of the local area of ​​the image to find out the points with large changes or relatively stable as candidate key points. The controller selects a group of representative key points from the candidate key points according to preset screening conditions, which include the number range, distribution range, and response value threshold of the key points, and outputs their coordinates (x, y) in the target image.

[0048] After obtaining the coordinates of the key points, the controller further determines two special key points located at the top and bottom of the target image, which are called the first key point and the second key point respectively. The first key point is located at the highest point of the boundary contour of the bottle, and the second key point is located at the lowest point of the boundary contour of the bottle, or the point on the boundary contour closest to the upper and lower edges of the image. The controller compares the vertical coordinates y of the key points and finds the point with the smallest y value as the first key point and the point with the largest y value as the second key point. If there are multiple highest or lowest points with the same y value, the average value of their horizontal coordinates x is taken as the x coordinate of the corresponding key point.

[0049] Next, the controller calculates the angle between the line connecting the first key point and the second key point and the horizontal direction as the angle of the target image. Specifically, the controller first calculates the slope k of the line connecting the two key points, that is, (y2-y1) / (x2-x1), where (x1, y1) and (x2, y2) are the coordinates of the first key point and the second key point, respectively. The controller calculates the angle θ between the slope k and the horizontal direction, that is, θ=arctan(k), in radians. Finally, the controller converts the angle θ into an angle value, that is, θ*180 / π, in degrees. The angle value represents the horizontal rotation angle of the target bottle in the target image, a positive value represents a clockwise rotation, and a negative value represents a counterclockwise rotation.

[0050] Similarly, the controller also calculates the angle between the line connecting the first key point and the second key point and the vertical direction as the inclination of the target image. Specifically, the controller first calculates the slope k' between the line connecting the first key point and the second key point and the vertical direction, that is, (x2-x1) / (y2-y1). Then, the controller calculates the angle φ between it and the vertical direction based on the slope k', that is, φ=arctan(k'), in radians. Finally, the controller converts the angle φ into an angle value, that is, φ*180 / π, in degrees. This angle value represents the vertical inclination angle of the target image in the image, a positive value indicates a tilt to the right, and a negative value indicates a tilt to the left.

[0051] For example, the controller performs key point detection on an image of a beer bottle and obtains 10 candidate key points. After screening, the controller determines that the point (400, 100) located at the center of the bottle mouth is the first key point, and the point (380, 620) located at the center of the bottle bottom is the second key point. The controller calculates the horizontal angle of the line connecting the two key points to be -2.86°, indicating that the beer bottle is slightly rotated to the left in the image; the vertical angle of the line is calculated to be -87.44°, indicating that the beer bottle is slightly tilted to the left in the image. The controller saves these two angle values ​​for subsequent image correction steps.

[0052] Step S104: Correcting the target bottle according to the angle and the inclination to achieve a preset standard angle and a preset standard inclination, and obtaining a corrected standard image.

[0053] In step S104, the angle of the target bottle is compared with the preset standard angle to obtain the angle difference, and the inclination of the target bottle is compared with the preset standard inclination to obtain the inclination difference; an affine transformation matrix is ​​constructed according to the angle difference and the inclination difference; and the affine transformation matrix is ​​used to perform an affine transformation on the target bottle so that the angle of the target bottle reaches the preset standard angle and the inclination of the target bottle reaches the preset standard inclination, thereby obtaining a corrected standard image.

[0054] Specifically, the controller compares the angle of the target bottle with the preset standard angle, calculates the difference between the two, and obtains the angle difference. In the embodiment of the present application, the preset standard angle is preferably set to 0 degrees, indicating that the target bottle remains horizontal in the target image without any rotation. If the actual angle of the target image is a positive value, it means that the target bottle rotates clockwise and needs to be rotated to the left by a certain angle to reach the preset standard angle; if the actual angle is a negative value, it means that the bottle rotates counterclockwise and needs to be rotated to the right by a certain angle to reach the preset standard angle. The absolute value of the angle difference indicates the angle that needs to be rotated, and the positive and negative signs of the angle difference indicate the direction that needs to be rotated.

[0055] Similarly, the controller also compares the inclination of the target image with the preset standard inclination, calculates the difference between the two, and obtains the inclination difference. The preset standard inclination is also 0 degrees, indicating that the target bottle maintains a vertical direction in the target image without any inclination. If the actual inclination of the target image is a positive value, it means that the bottle is tilted to the right and needs to be tilted to the left at a certain angle to reach the preset standard inclination; if the actual inclination is a negative value, it means that the bottle is tilted to the left and needs to be tilted to the right at a certain angle to reach the preset standard inclination. The absolute value of the inclination difference indicates the angle that needs to be tilted, and the positive and negative signs of the inclination difference indicate the direction that needs to be tilted.

[0056] After obtaining the angle difference and the inclination difference, the controller constructs an affine transformation matrix according to the angle difference and the inclination difference for geometric correction of the target image. In the embodiment of the present application, the affine transformation can be understood as linear transformation and translation of the pixel coordinates of the target bottle, thereby realizing the rotation, scaling, and tilting operations of the target bottle. The affine transformation matrix is ​​a 2×3 matrix, in which the first two columns represent the linear transformation coefficients and the last column represents the translation amount. In the embodiment of the present application, the formula of the affine transformation matrix is Among them, A is the affine transformation matrix, θ is the angle difference, φ is the tilt difference, t x is the translation distance in the x direction, t y is the translation distance in the y direction.

[0057] The controller uses the affine transformation matrix to perform affine transformation on the target bottle to obtain the corrected standard image. Specifically, the controller traverses each pixel point of the target bottle, calculates the transformed new coordinates (x', y') according to its coordinates (x, y) and the affine transformation matrix, and then fills the pixel values ​​of the new coordinates into the corresponding positions of the standard image. In this way, the controller can make the angle of the target bottle reach the preset standard angle and the inclination reach the preset standard inclination by performing linear transformation and translation on the coordinates of each pixel point.

[0058] Step S105: Based on the target bottle, feature extraction is performed on the standard image to obtain target features.

[0059] In step S105, the controller extracts features from the corrected standard image to obtain a set of numerical vectors that can accurately describe the visual attributes of the bottle, called target features. The controller uses a region-based feature extraction method to first segment the bottle image into multiple local regions, such as the bottle cap, bottle body, and bottle bottom, and then extracts color, texture, and shape from each local region. Finally, the features of all local regions are spliced ​​into a long vector as the target feature of the target bottle.

[0060] Step S106: Compare the target feature with the preset feature base library. If it is determined that the similarity between the target feature and the target base library feature is greater than or equal to the preset similarity threshold, determine the target bottle category corresponding to the target base library feature, and use the target bottle category as the bottle category corresponding to the bottle to be identified. The preset feature base library includes multiple base library features, and the target base library feature is any one of the multiple base library features.

[0061] In step S106, the controller compares the extracted target features with the base features in the preset feature base library, finds out the most similar base features, and thus identifies the specific category of the target image. The preset feature base library is a data set containing multiple base features, each of which corresponds to a known bottle category. The controller uses the KNN (K nearest neighbor) algorithm to compare and identify the features. Specifically, the controller traverses each base feature vector in the preset feature base library, calculates the cosine similarity between it and the target feature vector, and selects the target base feature vector with a similarity greater than a preset similarity threshold as a candidate matching result. Next, the controller votes and sorts the candidate matching results. Each candidate matching result corresponds to a target bottle category, and the controller counts the frequency of occurrence of each target bottle category, and takes the category with the highest frequency as the final recognition result. If a tie occurs, the controller selects the result with the highest similarity. The controller uses the matched target bottle category as the recognition result of the bottle to be identified, thereby completing the classification and recognition process of the bottle to be identified.

[0062] In a possible implementation, after step S106, the method further includes: if the target bottle category is a recyclable bottle category, then importing the target bottle into a recycling station; if the target bottle category is a non-recyclable bottle category, then importing the target bottle into a garbage station.

[0063] Specifically, the controller pre-establishes a mapping table of bottle categories and recycling attributes, which specifies which bottles of the target bottle category are recyclable and which are non-recyclable. Recyclable bottles generally include glass bottles, metal cans, and plastic bottles. Non-recyclable bottles generally include ceramic bottles and composite bottles. The controller marks each leaf node of the target bottle category as recyclable or non-recyclable to form a complete mapping table. When the target bottle is identified as a recyclable item, the controller will control the sorting equipment to import the target bottle into the recycling bin. When the target bottle is identified as a non-recyclable item, the controller will control the sorting equipment to import the target bottle into the garbage bin.

[0064] Reference Figure 2The present application also provides a target classification device based on image analysis, which is a controller, and the controller includes an acquisition module 201 and a processing module 202; the acquisition module 201 is used to acquire a target image containing a bottle to be identified; the processing module 202 is used to perform target detection on the target image to obtain a target bottle and extract a boundary contour of the target bottle; the processing module 202 is also used to determine the angle and inclination of the target bottle based on the boundary contour; the processing module 202 is also used to correct the target bottle according to the angle and inclination to achieve a preset standard angle and a preset standard inclination, and obtain a corrected standard image; the processing module 202 is also used to extract features from the standard image based on the target bottle to obtain target features; the processing module 202 is also used to compare the target feature with a preset feature base library, if it is determined that the similarity between the target feature and the target base library feature is greater than or equal to a preset similarity threshold, then determine the target bottle category corresponding to the target base library feature, and use the target bottle category as the bottle category corresponding to the bottle to be identified, the preset feature base library includes multiple base library features, and the target base library feature is any one of the multiple base library features.

[0065] In a possible implementation, the processing module 202 determines the angle and inclination of the target bottle based on the boundary contour, specifically including: the processing module 202 performs key point detection on the target bottle to obtain the key point coordinates of the target bottle; the processing module 202 determines the first key point and the second key point from the key point coordinates, the first key point is located at the top of the target bottle, and the second key point is located at the bottom of the target bottle; the processing module 202 calculates a first angle between a line connecting the first key point and the second key point and a horizontal direction, and determines the first angle as the angle of the target bottle; the processing module 202 calculates a second angle between a line connecting the first key point and the second key point and a vertical direction, and determines the second angle as the inclination of the target bottle.

[0066] In a possible implementation, the processing module 202 corrects the target bottle according to the angle and the inclination to achieve a preset standard angle and a preset standard inclination, and obtains a corrected standard image, specifically including: the processing module 202 compares the angle of the target bottle with the preset standard angle to obtain the angle difference, and compares the inclination of the target bottle with the preset standard inclination to obtain the inclination difference; the processing module 202 constructs an affine transformation matrix according to the angle difference and the inclination difference; the processing module 202 performs an affine transformation on the target bottle using the affine transformation matrix, so that the angle of the target bottle reaches the preset standard angle, the inclination of the target bottle reaches the preset standard inclination, and the corrected standard image is obtained.

[0067] In a possible implementation, before the acquisition module 201 acquires the target image containing the bottle to be identified, the method further includes: the acquisition module 201 acquires multiple initial images; the processing module 202 performs data amplification on the initial image to obtain an amplified image, where the data amplification includes at least one of rotation, scaling, cropping, flipping and color transformation; the processing module 202 uses the amplified image as a training sample, and uses a preset convolution model to train the amplified image to obtain a preset feature base library.

[0068] In a possible implementation, the processing module 202 uses the amplified image as a training sample, and uses a preset convolution model to train the amplified image to obtain a preset feature base library, specifically including: the processing module 202 uses the preset convolution model to extract features from the amplified image to obtain multiple training features; the processing module 202 performs cluster analysis on the multiple training features to obtain multiple cluster centers, and uses the multiple cluster centers as base library features in the preset feature base library.

[0069] In a possible implementation, the processing module 202 performs target detection on the target image, obtains the target bottle, and extracts the boundary contour of the target bottle, specifically including: the processing module 202 uses a preset target detection algorithm to perform target detection on the target image to obtain the position information of the target bottle; the processing module 202 extracts the image area of ​​the target bottle from the target image according to the position information; the processing module 202 uses a preset instance segmentation model to perform instance segmentation on the image area of ​​the target bottle to obtain the boundary contour of the target bottle.

[0070] In a possible implementation, the target bottle category includes a recyclable bottle category and a non-recyclable bottle category. After the processing module 202 uses the target bottle category as the bottle category corresponding to the bottle to be identified, the method further includes: if the target bottle category is a recyclable bottle category, the processing module 202 imports the bottle to be identified into a recycling station; if the target bottle category is a non-recyclable bottle category, the processing module 202 imports the bottle to be identified into a garbage station.

[0071] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0072] The present application also provides an electronic device. Figure 3 , Figure 3The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0073] The communication bus 302 is used to realize the connection and communication between these components.

[0074] The user interface 303 may include a display screen (Display) and a camera (Camera). The optional user interface 303 may also include a standard wired interface and a wireless interface.

[0075] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0076] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0077] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a target classification method based on image analysis.

[0078] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the application program stored in the memory 305 for a target classification method based on image analysis, and when executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0079] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments.

[0080] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0085] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0086] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A target classification method based on image analysis, characterized in that: The method comprises: Obtain a target image containing a bottle to be identified; Performing target detection on the target image to obtain a target bottle, and extracting a boundary contour of the target bottle; Based on the boundary contour, determining the angle and inclination of the target bottle; According to the angle and the inclination, the target bottle is corrected to reach a preset standard angle and a preset standard inclination, and a corrected standard image is obtained; Based on the target bottle, extracting features from the standard image to obtain target features; The target feature is compared with a preset feature base library. If it is determined that the similarity between the target feature and the target base library feature is greater than or equal to a preset similarity threshold, the target bottle category corresponding to the target base library feature is determined, and the target bottle category is used as the bottle category corresponding to the bottle to be identified. The preset feature base library includes multiple base library features, and the target base library feature is any one of the multiple base library features.

2. The method according to claim 1, characterized in that Determining the angle and inclination of the target bottle based on the boundary contour specifically includes: Perform key point detection on the target bottle to obtain the key point coordinates of the target bottle; Determining a first key point and a second key point from the key point coordinates, wherein the first key point is located at the top of the target bottle, and the second key point is located at the bottom of the target bottle; Calculating a first angle between a line connecting the first key point and the second key point and a horizontal direction, and determining the first angle as an angle of the target bottle; A second angle between a line connecting the first key point and the second key point and a vertical direction is calculated, and the second angle is determined as the inclination of the target bottle.

3. The method according to claim 1, characterized in that The method of correcting the target bottle according to the angle and the inclination to achieve a preset standard angle and a preset standard inclination, and obtaining a corrected standard image, specifically includes: Comparing the angle of the target bottle with a preset standard angle to obtain an angle difference, and comparing the inclination of the target bottle with a preset standard inclination to obtain an inclination difference; constructing an affine transformation matrix according to the angle difference and the inclination difference; The target bottle is affine transformed by using the affine transformation matrix, so that the angle of the target bottle reaches a preset standard angle, and the inclination of the target bottle reaches a preset standard inclination, thereby obtaining a corrected standard image.

4. The method according to claim 1, characterized in that: Before acquiring the target image containing the bottle to be identified, the method further includes: Acquire multiple initial images; Performing data amplification on the initial image to obtain an amplified image, wherein the data amplification includes at least one of rotation, scaling, cropping, flipping and color conversion; The amplified image is used as a training sample, and a preset convolution model is used to train the amplified image to obtain the preset feature base library.

5. The method according to claim 4, characterized in that The method of using the amplified image as a training sample and using a preset convolution model to train the amplified image to obtain the preset feature base library specifically includes: Using the preset convolution model to extract features from the amplified image to obtain a plurality of training features; Cluster analysis is performed on the plurality of training features to obtain a plurality of cluster centers, and the plurality of cluster centers are used as base database features in the preset feature base database.

6. The method according to claim 1, characterized in that The performing target detection on the target image to obtain a target bottle and extracting a boundary contour of the target bottle specifically includes: Using a preset target detection algorithm to perform target detection on the target image to obtain the position information of the target bottle; extracting an image region of the target bottle from the target image according to the position information; A preset instance segmentation model is used to perform instance segmentation on the image region of the target bottle to obtain a boundary contour of the target bottle.

7. The method according to claim 1, characterized in that The target bottle category includes a recyclable bottle category and a non-recyclable bottle category. After taking the target bottle category as the bottle category corresponding to the bottle to be identified, the method further includes: If the target bottle category is a recyclable bottle category, the bottle to be identified is introduced into a recycling station; If the target bottle category is a non-recyclable bottle category, the bottle to be identified is introduced into a garbage station.

8. A target classification device based on image analysis, characterized in that: The device comprises an acquisition module (201) and a processing module (202), wherein: The acquisition module (201) is used to acquire a target image containing a bottle to be identified; The processing module (202) is used to perform target detection on the target image, obtain a target bottle, and extract a boundary contour of the target bottle; The processing module (202) is further used to determine the angle and inclination of the target bottle based on the boundary contour; The processing module (202) is further used to calibrate the target bottle according to the angle and the inclination to achieve a preset standard angle and a preset standard inclination, and obtain a calibrated standard image; The processing module (202) is further used to extract characteristics of the standard image based on the target bottle to obtain target features; The processing module (202) is further configured to compare the target feature with a preset feature base library; if it is determined that the similarity between the target feature and the target base library feature is greater than or equal to a preset similarity threshold, then determine the target bottle category corresponding to the target base library feature, and use the target bottle category as the bottle category corresponding to the bottle to be identified; the preset feature base library includes a plurality of base library features, and the target base library feature is any one of the plurality of base library features.

9. An electronic device, characterized in that: The electronic device (300) comprises a processor (301), a memory (305), a user interface (303) and a network interface (304), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device (300) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images