Method for recognizing fruit ripeness, recognition device, control device, and storage medium
By combining and fusing visual and tactile information in depth, a combined feature vector is generated, which solves the problems of subjectivity and environmental adaptability in traditional fruit ripeness identification and achieves higher identification accuracy and reliability.
Patent Information
- Application Number
- CN202610492102.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-14
- Publication Date
- 2026-07-14
AI Technical Summary
Traditional fruit ripeness identification relies on human sensory judgment, which is highly subjective, has large identification errors, and is inefficient. Computer vision-based methods have low accuracy in complex environments.
By combining visual and tactile information, key features are extracted through a feature analysis network, and deep analysis is performed using a feature fusion network to generate a combined feature vector. This vector is then input into a maturity prediction network to predict and determine the maturity of the fruit.
It improves the accuracy and reliability of fruit ripeness identification, enabling precise identification in complex environments.
Smart Images
Figure CN122391711A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of fruit identification technology, and in particular to a method, identification device, control device, and storage medium for identifying fruit ripeness. Background Technology
[0002] Fruit ripeness is a key indicator of fruit quality, and its accurate identification is crucial for various stages of the fruit industry chain, including harvesting, grading, and sales. Traditional methods of fruit ripeness identification rely heavily on human sensory judgment, which is highly subjective, prone to errors, and inefficient.
[0003] With the rapid development of smart agriculture, artificial intelligence technologies such as computer vision and deep learning have been gradually applied to the field of fruit ripeness recognition. These technologies determine the ripeness of fruit by analyzing fruit images. However, these technologies are not adaptable to complex environments, especially in greenhouses or picking sites, where they are easily affected by factors such as changes in light, obstruction by objects, and dirt on the fruit surface, resulting in low accuracy in fruit ripeness recognition. Summary of the Invention
[0004] The main objective of this application is to provide a method, device, control device, and storage medium for identifying fruit maturity, with the aim of improving the accuracy of fruit maturity identification.
[0005] Firstly, this application provides a method for identifying fruit ripeness, including: Acquire visual and tactile information of the fruit to be identified; The visual information and the tactile information are input into a preset feature analysis network for feature analysis processing to obtain the visual feature vector corresponding to the visual information and the tactile feature vector corresponding to the tactile information. The visual feature vector and the tactile feature vector are input into a preset feature fusion network for feature fusion processing to obtain the combined feature vector corresponding to the fruit to be identified. The combined feature vector is input into a preset maturity prediction network for prediction processing to obtain the maturity probability distribution data corresponding to the fruit to be identified. This maturity probability distribution data is used to characterize the probability that the fruit to be identified is at different maturity levels. Based on the maturity probability distribution data, the maturity of the fruit to be identified is determined.
[0006] Secondly, this application also provides an identification device, the identification device comprising: A robotic arm used to pick up fruit; A camera device for acquiring visual information about the fruit; A tactile sensor is used to acquire tactile information about the fruit; The control unit, connected to the robotic arm, the imaging device, and the tactile sensor, is used to implement the fruit ripeness identification method described above. Thirdly, this application also provides a control device, which includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the fruit ripeness identification method as described above.
[0007] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the fruit ripeness identification method as described above.
[0008] This application provides a method, device, control device, and storage medium for identifying fruit maturity. By deeply analyzing and fusing the visual and tactile information of the fruit to be identified, a combined feature vector is obtained that can more comprehensively characterize the intrinsic features of the maturity of the fruit to be identified. Based on this combined feature vector, the maturity of the fruit to be identified under various environments can be accurately identified, significantly improving the accuracy and reliability of maturity identification. Attached Figure Description
[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 This is a flowchart illustrating the steps of a fruit ripeness identification method provided in an embodiment of this application. Figure 2 for Figure 1 A flowchart illustrating the sub-steps of the fruit ripeness identification method; Figure 3 A schematic block diagram of an identification device provided in an embodiment of this application; Figure 4 This is a schematic block diagram of a control device provided in an embodiment of this application.
[0011] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0013] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the described order. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0014] This application provides a method, device, control device, and storage medium for identifying fruit maturity. The fruit maturity identification method can be applied to either the identification device or the control device. The control device can be a terminal device or a server. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. The following explanation uses the application of the fruit maturity identification method to a server as an example.
[0015] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0016] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a fruit ripeness identification method provided in an embodiment of this application.
[0017] like Figure 1 As shown, the fruit maturity identification method includes steps S101 to S105.
[0018] Step S101: Obtain visual and tactile information of the fruit to be identified.
[0019] Visual information is used to characterize the appearance features of the fruit to be identified, such as color, texture, and shape, and can be image data acquired from different angles by a camera (e.g., a high-definition camera). Tactile information is used to characterize the tactile features of the fruit to be identified, such as hardness and elasticity, and can be pressure distribution data, surface deformation data, hardness data, etc., acquired by a tactile sensor.
[0020] It should be noted that the collection of visual and tactile information should be completed simultaneously to ensure consistency between the two in the time dimension, and to avoid the accuracy of subsequent recognition results being affected by changes in the state of the fruit due to excessively long collection intervals.
[0021] For example, assuming the fruit to be identified is a kiwi, a robotic arm, in conjunction with a tactile sensor and an imaging device, collects tactile and visual information about the kiwi. Specifically, the robotic arm selects four points at the equator of the kiwi for gripping. During gripping, the grippers on the robotic arm apply a fixed force to ensure consistent contact force each time, reducing external interference. When the grippers are stationary, the tactile sensor and imaging device simultaneously record the tactile and visual images of the kiwi, thus obtaining the tactile and visual information of the kiwi.
[0022] Understandably, in some cases, visual information is sufficient to determine the ripeness stage of fruit. For example, as ripeness increases, unripe fruit that is initially a greenish hue will gradually transform into specific ripe colors such as yellow and red. The skin texture may change from smooth to relatively rough or exhibit specific patterns, and the roundness of the fruit's shape may increase. However, in other cases, relying solely on visual information is insufficient to accurately determine fruit ripeness. For instance, some fruits do not show obvious changes in appearance during ripening, or the visual information collection process may be affected by external factors such as lighting, background, or stains or blemishes on the fruit's surface. In these situations, relying solely on visual information can easily lead to misjudgment. Furthermore, some fruits may exhibit ripeness characteristics on the outside, but their internal ripeness indicators, such as sweetness and firmness, may not have reached the ideal state. Therefore, by combining visual and tactile information, and fully utilizing their complementarity, a comprehensive assessment of fruit ripeness can be conducted from multiple dimensions, thereby improving the accuracy and reliability of fruit ripeness identification.
[0023] In one embodiment, a robotic arm picks up a fruit and places it at a set position. A camera takes a picture of the fruit at the set position to obtain visual information of the fruit to be identified. A tactile sensor also acquires tactile information of the fruit at the set position. The acquired visual and tactile information of the fruit to be identified is then sent to a control unit for processing.
[0024] Step S102: Input the visual information and tactile information into a preset feature analysis network for feature analysis processing to obtain the visual feature vector corresponding to the visual information and the tactile feature vector corresponding to the tactile information.
[0025] The feature analysis network is used to extract key features related to fruit ripeness from the input data, thereby obtaining enhanced feature vectors with higher discriminativeness and distinguishability. Specifically, for visual information, the feature analysis network performs feature analysis and outputs visual feature vectors that can be used to characterize key visual features related to fruit ripeness, such as the distribution / gradient of the peel color and texture details of the fruit to be identified; for tactile features, the feature analysis network performs feature analysis and outputs tactile feature vectors that can be used to characterize key tactile features related to fruit ripeness, such as the surface hardness and elastic modulus of the fruit to be identified.
[0026] Understandably, through the processing of feature analysis networks, it is possible to effectively remove interfering factors from visual and tactile information, retain key features closely related to maturity in visual and tactile information, and help improve the accuracy of subsequent feature fusion and maturity recognition.
[0027] In one embodiment, the feature analysis network includes a feature extraction subnetwork and a feature transformation subnetwork.
[0028] The feature extraction subnetwork is used to initially extract basic features from the input visual and tactile information, while the feature transformation network is used to further map and transform the basic features output by the feature extraction subnetwork into more discriminative feature vectors.
[0029] In one embodiment, such as Figure 2 As shown, step S102 includes sub-steps S1021 to S1023.
[0030] Sub-step S1021: Input visual information and tactile information into the feature extraction sub-network for feature extraction processing to obtain first visual feature information and first tactile feature information.
[0031] The feature extraction subnetwork can be a convolutional neural network (CNN) structure, which performs progressive downsampling and feature extraction on image data through multiple convolutional and pooling layers to capture basic visual features such as color channel information, local texture patterns, and edge contours of the fruit peel.
[0032] For example, assuming the feature extraction subnetwork is a convolutional neural network (CNN), the feature extraction layer of the CNN is used to capture low-level features of visual information and tactile information respectively, to obtain the first visual feature information corresponding to the visual information and the first tactile feature information corresponding to the tactile information.
[0033] Sub-step S1022: Input the first visual feature information and the first tactile feature information into the feature transformation sub-network to determine the visual weight vector corresponding to the first visual feature information and the tactile weight vector corresponding to the first tactile feature information; and perform feature transformation processing on the first visual feature information and the first tactile feature information based on the visual weight vector and the tactile weight vector respectively to obtain the visual feature vector and the tactile feature vector.
[0034] For example, suppose the feature transformation subnetwork is a deep neural network structure composed of multiple residual blocks. Each residual block includes a convolutional layer and an attention module that introduces an attention mechanism. By utilizing the attention module to adaptively focus on key features related to fruit ripeness in the basic features, the final output feature vector can effectively improve its ability to represent ripeness differences.
[0035] In one embodiment, the feature transformation subnetwork includes a pooling layer, a fully connected layer, and an output layer. The pooling layer can be used to reduce the dimensionality of the feature information, thereby reducing redundant information and retaining more key features related to fruit ripeness. For example, through max pooling or average pooling operations, high-dimensional first visual and first tactile feature information can be mapped to a lower-dimensional feature space. The fully connected layer can receive the compressed first visual and first tactile feature information output from the pooling layer, and generate weight vectors representing the importance of channels by learning the dependencies between channels in the first visual and first tactile feature information. The output layer is used to weight the features of each channel according to the weight vector output by the fully connected layer, and output the fused feature information.
[0036] In one embodiment, the first visual feature information and the first tactile feature information are input into a feature transformation subnetwork to determine the visual weight vector corresponding to the first visual feature information and the tactile weight vector corresponding to the first tactile feature information. This includes: performing average pooling processing on the visual feature information and the tactile feature information respectively through a pooling layer to obtain the visual feature descriptor corresponding to the visual feature information and the tactile feature descriptor corresponding to the tactile feature information; and processing the visual feature descriptor and the tactile feature descriptor respectively through a fully connected layer to determine the weight of each visual element in the visual feature information and the weight of each tactile element in the tactile feature information, thereby obtaining the visual weight vector corresponding to the visual feature information and the tactile weight vector corresponding to the tactile feature information.
[0037] In one embodiment, feature transformation processing is performed on the first visual feature information and the first tactile feature information based on the visual weight vector and the tactile weight vector, respectively, to obtain the visual feature vector and the tactile feature vector. This includes: performing weighted fusion processing on the visual feature information and the visual weight vector, and the tactile feature information and the tactile weight vector, respectively, through the output layer to obtain the visual feature vector corresponding to the visual information and the tactile feature vector corresponding to the tactile information.
[0038] For example, suppose the feature transformation subnetwork processes feature information in three stages: Squeeze, Excitation, and Scale. These three stages are performed by the pooling layer, fully connected layer, and output layer of the feature transformation subnetwork, respectively. Suppose that both the first visual feature information and the first tactile feature information are represented by feature maps. Input feature transformation subnetwork. Specifically, the processing of the first visual feature information and the first tactile feature information is as follows: First, in the Squeeze stage, the input feature map is processed... Perform a global average pooling operation. This operation aggregates spatial dimensional information channel by channel, averaging the pixel values of each channel to generate a channel descriptor. Each element captures a global statistical description of the corresponding channel, achieving effective compression of spatial information.
[0039] Secondly, in the Excitation stage, the non-linear dependencies between the channels are learned using the two fully connected layers to generate adaptive weights.
[0040] Finally, in the scaling phase, the weights are... Broadcasting is performed through element-wise multiplication and applied to the original feature map. This enables channel-level feature recalibration, where channels with higher weights are enhanced to highlight key information, while channels with lower weights are suppressed to reduce noise interference.
[0041] Understandably, by processing the first visual and first tactile feature information through the feature transformation subnetwork, it is possible to adaptively learn and highlight the channel features in the first visual and first tactile feature information that are more critical to fruit ripeness identification. The final output is a visual feature vector that has stronger discriminative ability than the first visual and first tactile feature information and can focus more on the core information reflecting the fruit ripeness status, as well as a tactile feature vector corresponding to the tactile information, which helps to improve the accuracy of fruit ripeness identification.
[0042] In one embodiment, determining the weights of each visual element in the visual feature information and the weights of each tactile element in the tactile feature information based on visual feature descriptors and tactile feature descriptors respectively, to obtain a visual weight vector corresponding to the visual feature information and a tactile weight vector corresponding to the tactile feature information, includes: compressing the visual feature descriptors and tactile feature descriptors respectively to obtain a first visual compressed feature vector and a first tactile compressed feature vector; performing nonlinear transformation processing on the first visual compressed feature vector and the first tactile compressed feature vector respectively using a preset activation function to obtain a second visual compressed feature vector and a second tactile compressed feature vector; and performing dimension restoration processing on the second visual compressed feature vector and the second tactile compressed feature vector respectively to obtain a visual weight vector corresponding to the visual feature information and a tactile weight vector corresponding to the tactile feature information.
[0043] For example, suppose the fully connected layer includes a dimensionality reduction fully connected layer and an dimensionality increase fully connected layer. First, the dimensionality reduction fully connected layer compresses the visual and tactile feature descriptors, reducing their channel count from C to a first visual compressed feature vector and a first tactile compressed feature vector with a channel count of C / r (where r is the reduction ratio), thus achieving feature dimensionality reduction and initial extraction of key information. Next, the ReLU activation function is used to apply a non-linear transformation, enhancing the model's ability to express complex feature relationships by introducing non-linear factors, resulting in a second visual compressed feature vector and a second tactile compressed feature vector with a channel count of C / r. Finally, the dimensionality increase fully connected layer restores the dimensions of the second visual compressed feature vector and the second tactile compressed feature vector from the channel count C / r back to the original channel count C. Finally, the Sigmoid activation function is used to output the normalized visual weight vectors corresponding to the second visual compressed feature vector and the second tactile compressed feature vector, respectively. and tactile weight vector Each element in the visual weight vector and tactile weight vector represents the importance of each channel feature in the corresponding visual feature descriptor and tactile feature descriptor, respectively. The larger the value, the higher the contribution of the channel feature in the fruit ripeness recognition process.
[0044] Understandably, the visual weight vectors and tactile weight vectors obtained by compressing, nonlinearly transforming, and restoring the visual and tactile feature descriptors can help the model more accurately focus on the visual and tactile feature channels that have a key impact on fruit ripeness recognition, thereby helping to improve the accuracy of fruit ripeness recognition.
[0045] Step S103: Input the visual feature vector and tactile feature vector into the preset feature fusion network for feature fusion processing to obtain the combined feature vector corresponding to the fruit to be identified.
[0046] Feature fusion can be achieved through weighted fusion, splicing fusion, or attention-based fusion. It is understandable that by deeply integrating complementary visual and tactile features, the correlation information between different modalities can be fully explored, avoiding potential identification biases that may arise from single-modal features in maturity recognition. The resulting combined feature vector can characterize the fruit's maturity state from more dimensions, aiding in accurate subsequent judgments of fruit maturity.
[0047] In one embodiment, the feature fusion network includes a feature enhancement subnetwork and a feature fusion subnetwork. Visual feature vectors and tactile feature vectors are input into a preset feature fusion network for feature fusion processing to obtain a combined feature vector corresponding to the fruit to be identified. This includes: inputting the visual feature vectors and tactile feature vectors into the feature enhancement subnetwork respectively to perform feature enhancement processing on the visual feature vectors and tactile feature vectors respectively to obtain a target visual feature vector and a target tactile feature vector; and inputting the target visual feature vector and the target tactile feature vector into the feature fusion subnetwork for splicing and fusion processing to obtain a combined feature vector corresponding to the fruit to be identified.
[0048] Among them, the feature enhancement subnetwork can be used to perform bidirectional interactive enhancement on the feature vectors of two different input modalities to highlight the key information that is highly correlated with fruit maturity recognition in different modalities. The feature fusion subnetwork is used to splice and fuse the feature vectors of the two different modalities after bidirectional interactive enhancement along the dimension to obtain a combined feature vector that can more comprehensively and accurately reflect the fruit maturity information.
[0049] In one embodiment, feature enhancement processing is performed on the visual feature vector and the tactile feature vector respectively to obtain the target visual feature vector and the target tactile feature vector. This includes: calculating the visual contribution of the visual feature vector to the tactile feature vector, and using the visual contribution to perform feature enhancement processing on the tactile feature vector to obtain the target visual feature information; and calculating the tactile contribution of the tactile feature vector to the visual feature vector, and using the tactile contribution to perform feature enhancement processing on the tactile feature vector to obtain the target visual feature vector.
[0050] Among them, the visual contribution is used to characterize the degree of influence of fruit maturity-related information contained in the visual feature vector on the tactile feature vector, while the tactile contribution is used to characterize the degree of influence of fruit maturity-related information contained in the tactile feature vector on the visual feature vector.
[0051] For example, the feature enhancement subnetwork includes two independent attention branches, one for visual-tactile interaction and the other for tactile-visual interaction. Let the visual feature vector be... and tactile feature vectors are .
[0052] Specifically, firstly, using visual feature vectors As a query, tactile feature vectors Using these as keys and values, the tactile contribution of the tactile feature vector to the visual feature vector is calculated. The specific process is as follows: First, a linear transformation is used to map the visual feature vectors and tactile feature vectors to the same dimensional space, resulting in a query matrix. Key matrix and value matrix ,in It is a learnable projection matrix.
[0053] Then, calculate the transpose of the query matrix and the key matrix. The dot product, and the result divided by the square root of the feature dimension ( ,in The feature dimensions of the extracted modality are scaled and then processed by the Softmax function to obtain the attention weight matrix. This attention weight matrix is the tactile contribution degree used to characterize the degree of contribution of the tactile feature vector to each feature component in the visual feature vector.
[0054] Finally, the attention weight matrix and the value matrix are weighted and summed to obtain the target visual feature information enhanced by tactile contribution. .
[0055] The calculation formulas corresponding to the above process are as follows:
[0056] Similarly, in the haptic-visual interaction branch, haptic feature vectors are used... As a query, visual feature vectors Using visual feature vectors as keys and values, the visual contribution of the visual feature vector to the tactile feature vector is calculated, ultimately yielding the target tactile feature information enhanced by the visual contribution. For details, please refer to the above description of target visual feature information. The calculation process will not be elaborated here.
[0057] Understandably, by calculating the contribution of vision and touch in both directions and enhancing features, the feature vectors of the two modes can fully absorb information from the other mode that is more useful for maturity recognition while retaining their own core feature information, thus achieving deep interaction and enhancement of features.
[0058] Step S104: Input the combined feature vector into the preset maturity prediction network for prediction processing to obtain the maturity probability distribution data of the fruit to be identified.
[0059] Among them, the maturity prediction network can be a classification network built based on deep learning, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or a combination of them. The maturity prediction network is used to perform deep feature extraction and nonlinear mapping on the combined feature vectors, and transform the fused multimodal information into a probability distribution representation of fruit maturity.
[0060] In one embodiment, the maturity prediction network includes a feature transformation subnetwork and a feature conversion subnetwork. The process of inputting a combined feature vector into the preset maturity prediction network for prediction processing to obtain the maturity probability distribution data corresponding to the fruit to be identified includes: inputting the combined feature vector into the feature transformation subnetwork for at least one nonlinear transformation to obtain a target combined feature vector; and inputting the target combined feature vector into the feature conversion subnetwork for normalization processing to obtain the maturity probability distribution data.
[0061] The feature transformation subnetwork can adopt a multilayer perceptron (MLP) structure, including multiple fully connected layers (FC), each followed by a ReLU activation function, to perform non-linear mapping on the input combined feature vector to uncover deeper feature associations. The feature transformation subnetwork then uses the Softmax function as the activation function to normalize the target combined feature vector output by the feature transformation subnetwork, converting it into probability values for each maturity category. The sum of all probability values is 1, clearly showing the likelihood of the fruit to be identified being at different maturity levels.
[0062] Step S105: Determine the maturity of the fruit to be identified based on the maturity probability distribution data.
[0063] Specifically, one could determine the maturity category corresponding to the highest probability value from the maturity probability distribution data, and then identify that category as the maturity level of the fruit to be identified.
[0064] For example, suppose maturity includes three categories: immature, mature, and overripe. If the probability distribution data of the maturity of the fruit to be identified has a probability value of 0.75 for "mature", 0.15 for "immature", and 0.10 for "overripe", then the "mature" category with the highest probability value is selected as the maturity of the fruit.
[0065] In some embodiments, determining the maturity of the fruit to be identified based on the maturity probability distribution data further includes: if the probability value corresponding to the maturity of the fruit to be identified is less than a preset threshold, then the fruit to be identified is determined as the fruit to be reviewed.
[0066] The specific value of the preset threshold can be adjusted according to the actual application scenario and the requirements for recognition accuracy. Fruits awaiting verification refer to fruits that require further inspection and recognition of their maturity, either manually or using the fruit maturity recognition method in any embodiment of this application. For example, if the preset threshold is 0.8, and the probability value corresponding to the "mature" category of a certain fruit is 0.75 (0.75 < 0.8), then the fruit is marked as a fruit awaiting verification. This allows for manual judgment by operators or a second recognition by the system using the fruit maturity recognition method, thereby improving the reliability of the final recognition result.
[0067] Understandably, by using a preset threshold to perform a secondary verification of the ripeness recognition results of the fruit to be identified, the risk of misjudgment that may be caused by the principle of maximizing a single probability value can be effectively reduced.
[0068] In some embodiments, the training process of the maturity prediction network includes: acquiring visual and tactile information corresponding to multiple sample fruits, and acquiring maturity level data corresponding to each sample fruit; inputting the visual and tactile information corresponding to multiple sample fruits into a preset feature analysis network for feature analysis processing to obtain visual feature vectors and tactile feature vectors corresponding to each visual information; inputting the visual and tactile feature vectors corresponding to multiple sample fruits into a preset feature fusion network for feature fusion processing to obtain combined feature vectors corresponding to each sample fruit; inputting the combined feature vectors corresponding to multiple sample fruits into the maturity prediction network for prediction processing to obtain maturity probability distribution data corresponding to each sample fruit, wherein the maturity probability distribution data is used to characterize the probability that the sample fruit belongs to different maturity levels; and updating the parameters of the maturity prediction network based on the maturity level data and maturity probability distribution data corresponding to multiple sample fruits until the maturity prediction network converges.
[0069] During the training process of the maturity prediction network, the processing of visual and tactile information corresponding to multiple sample fruits by the feature analysis network, feature fusion network, and maturity prediction network can be referred to the above embodiments, and will not be repeated here.
[0070] It should be noted that, to ensure data diversity and representativeness, multiple sample fruits should cover samples from different maturity stages. Furthermore, for multiple images serving as visual and tactile information, all images can be uniformly scaled to ensure consistent image size input to the feature analysis network. Additionally, random cropping of each image can enhance sample information diversity and prevent model overfitting. Further, data augmentation operations such as random rotation, flipping, scaling, and translation can be applied to each image to simulate different acquisition environments and angles, effectively improving the model's robustness. Finally, a cross-entropy loss function can be introduced and combined with the Adam optimizer to update the parameters of the maturity prediction network, thereby optimizing its predictive performance.
[0071] The fruit maturity recognition method provided in the above embodiments deeply analyzes and fuses the visual and tactile information of the fruit to be identified, and obtains a combined feature vector that can more comprehensively characterize the intrinsic features of the maturity of the fruit to be identified. Based on the combined feature vector, the maturity of the fruit to be identified under various environments can be accurately identified, which significantly improves the accuracy and reliability of maturity recognition.
[0072] Please refer to Figure 3 , Figure 3 This is a schematic block diagram of an identification device provided in an embodiment of this application.
[0073] like Figure 3 As shown, the identification device 200 includes: Robotic arm 201 is used to pick up fruit; The imaging device 202 is used to acquire visual information about the fruit; Tactile sensor 203 is used to acquire tactile information about the fruit; The control unit 204 is connected to the robotic arm 201, the imaging device 202 and the tactile sensor 203, and is used to implement the fruit ripeness recognition method as described in any of the above embodiments.
[0074] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 4 The control device shown is running.
[0075] Please see Figure 4 , Figure 4 This is a schematic block diagram of a control device provided in an embodiment of this application.
[0076] like Figure 4As shown, the control device 300 includes a processor 301, a memory 302, and a network interface 303 connected via a system bus. The memory 302 may include a storage medium and internal memory. The storage medium may be non-volatile or volatile.
[0077] The storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause the processor to perform any method for identifying fruit ripeness.
[0078] The processor 302 provides computing and control capabilities to support the operation of the entire control unit.
[0079] Internal memory provides an environment for the execution of computer programs stored in the storage medium. When these computer programs are executed by the processor, the processor can perform any method for recognizing fruit ripeness.
[0080] This network interface 303 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the control device 300 to which the present application is applied. The specific control device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0081] It should be understood that processor 301 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.
[0082] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Acquire visual and tactile information of the fruit to be identified; Visual and tactile information are input into a preset feature analysis network for feature analysis processing to obtain the visual feature vector corresponding to the visual information and the tactile feature vector corresponding to the tactile information. The visual feature vector and the tactile feature vector are input into a preset feature fusion network for feature fusion processing to obtain the combined feature vector corresponding to the fruit to be identified. The combined feature vectors are input into a pre-defined maturity prediction network for prediction processing, resulting in maturity probability distribution data for the fruit to be identified. This maturity probability distribution data characterizes the probability that the fruit to be identified is at different maturity levels. The maturity level of the fruit to be identified is determined based on the maturity probability distribution data.
[0083] In one embodiment, the feature analysis network includes a feature extraction subnetwork and a feature transformation subnetwork. When the processor inputs visual information and tactile information into the preset feature analysis network for feature analysis processing to obtain the visual feature vector corresponding to the visual information and the tactile feature vector corresponding to the tactile information, it is used to: Visual and tactile information are input into the feature extraction sub-network for feature extraction processing to obtain first visual feature information and first tactile feature information; The first visual feature information and the first tactile feature information are input into the feature transformation subnetwork to determine the visual weight vector corresponding to the first visual feature information and the tactile weight vector corresponding to the first tactile feature information; and Based on the visual weight vector and the tactile weight vector, feature transformation processing is performed on the first visual feature information and the first tactile feature information, respectively, to obtain the visual feature vector and the tactile feature vector.
[0084] In one embodiment, the feature transformation subnetwork includes a pooling layer and a fully connected layer. When the processor inputs the first visual feature information and the first tactile feature information into the feature transformation subnetwork to determine the visual weight vector corresponding to the first visual feature information and the tactile weight vector corresponding to the first tactile feature information, it is used to: The visual feature information and tactile feature information are processed by average pooling layer to obtain the visual feature descriptor corresponding to the visual feature information and the tactile feature descriptor corresponding to the tactile feature information. The visual feature descriptors and tactile feature descriptors are processed separately by a fully connected layer to determine the weights of each visual element in the visual feature information and the weights of each tactile element in the tactile feature information, thereby obtaining the visual weight vector corresponding to the visual feature information and the tactile weight vector corresponding to the tactile feature information.
[0085] In one embodiment, when the processor determines the weights of each visual element in the visual feature information and the weights of each tactile element in the tactile feature information based on the visual feature descriptor and the tactile feature descriptor, respectively, to obtain the visual weight vector corresponding to the visual feature information and the tactile weight vector corresponding to the tactile feature information, the processor is configured to: The visual feature descriptors and tactile feature descriptors are compressed separately to obtain the first visual compressed feature vector and the first tactile compressed feature vector. The first visual compression feature vector and the first tactile compression feature vector are subjected to nonlinear transformation processing using a preset activation function to obtain the second visual compression feature vector and the second tactile compression feature vector. The second visual compressed feature vector and the second tactile compressed feature vector are subjected to dimension restoration processing to obtain the visual weight vector corresponding to the visual feature information and the tactile weight vector corresponding to the tactile feature information.
[0086] In one embodiment, the feature fusion network includes a feature enhancement subnetwork and a feature fusion subnetwork. When the processor inputs visual feature vectors and tactile feature vectors into the preset feature fusion network for feature fusion processing to obtain a combined feature vector corresponding to the fruit to be identified, it is used to: The visual feature vector and the tactile feature vector are input into the feature enhancement sub-network respectively to perform feature enhancement processing on the visual feature vector and the tactile feature vector respectively, so as to obtain the target visual feature vector and the target tactile feature vector. The target visual feature vector and the target tactile feature vector are input into the feature fusion subnetwork for splicing and fusion processing to obtain the combined feature vector corresponding to the fruit to be identified.
[0087] In one embodiment, when the processor performs feature enhancement processing on the visual feature vector and the tactile feature vector respectively to obtain the target visual feature vector and the target tactile feature vector, it is used to implement: Calculate the visual contribution of the visual feature vector to the tactile feature vector, and use the visual contribution to perform feature enhancement processing on the tactile feature vector to obtain the target visual feature information; and Calculate the tactile contribution of the tactile feature vector to the visual feature vector, and use the tactile contribution to perform feature enhancement processing on the tactile feature vector to obtain the target visual feature vector.
[0088] In one embodiment, the maturity prediction network includes a feature transformation subnetwork and a feature conversion subnetwork, and the processor, in its implementation, is used to implement: The combined feature vectors are input into a pre-defined maturity prediction network for prediction processing to obtain the maturity probability distribution data corresponding to the fruit to be identified, including: The combined feature vector is input into the feature transformation sub-network for at least one nonlinear transformation to obtain the target combined feature vector. The target combination feature vector is input into the feature transformation sub-network for normalization processing to obtain mature probability distribution data.
[0089] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the control device described above can be referred to the corresponding process in the aforementioned embodiments of the fruit maturity identification method, and will not be repeated here.
[0090] The control device of this application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0091] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can refer to various embodiments of the fruit maturity identification method of this application.
[0092] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.
[0093] Furthermore, the computer's usable storage medium may primarily include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a given function, etc.; the data storage area may store data created based on the use of blockchain nodes, etc. The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.
[0094] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0095] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0096] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for identifying fruit ripeness, characterized in that, include: Acquire visual and tactile information of the fruit to be identified; The visual information and the tactile information are input into a preset feature analysis network for feature analysis processing to obtain the visual feature vector corresponding to the visual information and the tactile feature vector corresponding to the tactile information. The visual feature vector and the tactile feature vector are input into a preset feature fusion network for feature fusion processing to obtain the combined feature vector corresponding to the fruit to be identified. The combined feature vector is input into a preset maturity prediction network for prediction processing to obtain the maturity probability distribution data corresponding to the fruit to be identified. This maturity probability distribution data is used to characterize the probability that the fruit to be identified is at different maturity levels. Based on the maturity probability distribution data, the maturity of the fruit to be identified is determined.
2. The fruit ripeness identification method as described in claim 1, characterized in that, The feature analysis network includes a feature extraction subnetwork and a feature transformation subnetwork; The step of inputting the visual information and the tactile information into a preset feature analysis network for feature analysis processing to obtain the visual feature vector corresponding to the visual information and the tactile feature vector corresponding to the tactile information includes: The visual information and the tactile information are input into the feature extraction sub-network for feature extraction processing to obtain first visual feature information and first tactile feature information; The first visual feature information and the first tactile feature information are input into the feature transformation subnetwork to determine the visual weight vector corresponding to the first visual feature information and the tactile weight vector corresponding to the first tactile feature information; and Based on the visual weight vector and the tactile weight vector, feature transformation processing is performed on the first visual feature information and the first tactile feature information respectively to obtain the visual feature vector and the tactile feature vector.
3. The fruit ripeness identification method as described in claim 2, characterized in that, The feature transformation subnetwork includes a pooling layer and a fully connected layer; The step of inputting the first visual feature information and the first tactile feature information into the feature transformation sub-network to determine the visual weight vector corresponding to the first visual feature information and the tactile weight vector corresponding to the first tactile feature information includes: The visual feature information and the tactile feature information are subjected to average pooling processing by the pooling layer to obtain the visual feature descriptor corresponding to the visual feature information and the tactile feature descriptor corresponding to the tactile feature information. The fully connected layer processes the visual feature descriptor and the tactile feature descriptor respectively to determine the weight of each visual element in the visual feature information and the weight of each tactile element in the tactile feature information, thereby obtaining the visual weight vector corresponding to the visual feature information and the tactile weight vector corresponding to the tactile feature information.
4. The fruit ripeness identification method as described in claim 3, characterized in that, The step of determining the weights of each visual element in the visual feature information and the weights of each tactile element in the tactile feature information based on the visual feature descriptor and the tactile feature descriptor, respectively, to obtain the visual weight vector corresponding to the visual feature information and the tactile weight vector corresponding to the tactile feature information, includes: The visual feature descriptor and the tactile feature descriptor are compressed respectively to obtain a first visual compressed feature vector and a first tactile compressed feature vector; The first visual compression feature vector and the first tactile compression feature vector are subjected to nonlinear transformation processing using a preset activation function to obtain the second visual compression feature vector and the second tactile compression feature vector. The second visual compression feature vector and the second tactile compression feature vector are subjected to dimension restoration processing to obtain the visual weight vector corresponding to the visual feature information and the tactile weight vector corresponding to the tactile feature information.
5. The fruit ripeness identification method as described in claim 1, characterized in that, The feature fusion network includes a feature enhancement subnetwork and a feature fusion subnetwork; The step of inputting the visual feature vector and the tactile feature vector into a preset feature fusion network for feature fusion processing to obtain the combined feature vector corresponding to the fruit to be identified includes: The visual feature vector and the tactile feature vector are input into the feature enhancement sub-network respectively to perform feature enhancement processing on the visual feature vector and the tactile feature vector respectively, so as to obtain the target visual feature vector and the target tactile feature vector; The target visual feature vector and the target tactile feature vector are input into the feature fusion subnetwork for splicing and fusion processing to obtain the combined feature vector corresponding to the fruit to be identified.
6. The fruit ripeness identification method as described in claim 5, characterized in that, The step of performing feature enhancement processing on the visual feature vector and the tactile feature vector respectively to obtain the target visual feature vector and the target tactile feature vector includes: Calculate the visual contribution of the visual feature vector to the tactile feature vector, and use the visual contribution to perform feature enhancement processing on the tactile feature vector to obtain target visual feature information; and Calculate the tactile contribution of the tactile feature vector to the visual feature vector, and use the tactile contribution to perform feature enhancement processing on the tactile feature vector to obtain the target visual feature vector.
7. The fruit ripeness identification method according to any one of claims 1-6, characterized in that, The maturity prediction network includes a feature transformation subnetwork and a feature conversion subnetwork; The step of inputting the combined feature vector into a preset maturity prediction network for prediction processing to obtain the maturity probability distribution data corresponding to the fruit to be identified includes: The combined feature vector is input into the feature transformation sub-network for at least one nonlinear transformation process to obtain the target combined feature vector. The target combination feature vector is input into the feature transformation sub-network for normalization processing to obtain mature probability distribution data.
8. An identification device, characterized in that, The identification device includes: A robotic arm used to pick up fruit; A camera device for acquiring visual information about the fruit; A tactile sensor is used to acquire tactile information about the fruit; The control unit, connected to the robotic arm, the imaging device, and the tactile sensor, is used to implement the fruit ripeness recognition method as described in any one of claims 1 to 7.
9. A control device, characterized in that, The control device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the fruit ripeness identification method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the fruit ripeness identification method as described in any one of claims 1 to 7.