Cooperative control method and system based on Internet of Things and cloud computing, and electronic equipment

Abstract

The invention relates to the field of cooperative control, and particularly discloses a cooperative control method and system based on the Internet of Things and cloud computing and electronic equipment, and the force of a gripper is determined based on a shot image of a fruit by using the Internet of Things and the cloud computing technology. Therefore, the robot is used for picking operation in an autonomous collaborative operation mode, the picked fruits can be effectively prevented from being broken, and meanwhile the labor cost of picking operation is greatly reduced.

Description

Technical Field

[0001] The present application relates to the field of collaborative control, and more specifically to a collaborative control method, system and electronic device based on the Internet of Things and cloud computing. Background Art

[0002] Orchard picking refers to the activity of harvesting fruits in an orchard. This work usually requires a lot of manpower and needs to be done quickly when the fruit is ripe to ensure the freshness and quality of the fruit. Especially in the process of picking ripe fruits, it is often done by a lot of manpower. However, manual picking may cause damage to fruit trees and fruits, while consuming a lot of manpower costs and low picking efficiency. It is easy to miss the best ripening period of fruits due to long picking time. With the development of artificial intelligence technology, picking robots have appeared in people's sight, which can greatly reduce labor costs. However, in the operation of picking robots, there are still cases where the grasping force is too large or too small to pick rotten fruits, or the grasping force is too small to pick fruits.

[0003] Therefore, an optimized collaborative control method is expected. Summary of the invention

[0004] In order to solve the above technical problems, this application is proposed. The embodiments of this application provide a collaborative control method, system and electronic device based on the Internet of Things and cloud computing, which use the Internet of Things and cloud computing technology to determine the strength of the gripper based on the captured image of the fruit. In this way, using the robot autonomous collaborative operation mode to perform the picking operation can effectively prevent the picked fruits from breaking, and at the same time greatly reduce the labor cost of the picking operation.

[0005] According to one aspect of the present application, a collaborative control method based on the Internet of Things and cloud computing is provided, which includes:

[0006] Acquire a photographic image of the fruit;

[0007] Extracting and encoding the captured image of the fruit to obtain an optimized enhanced fruit feature vector;

[0008] The optimized fortified fruit feature vector is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripping force should be increased or decreased.

[0009] According to another aspect of the present application, a collaborative control system based on the Internet of Things and cloud computing is also provided, which includes:

[0010] A fruit image acquisition module is used to acquire the photographed image of the fruit;

[0011] An optimization module, used for extracting and encoding features of the captured image of the fruit to obtain an optimized enhanced fruit feature vector;

[0012] The gripping force control module is used to pass the optimized fortified fruit feature vector through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripping force should be increased or decreased.

[0013] According to another aspect of the present application, an electronic device is further provided, the electronic device comprising:

[0014] at least one processor;

[0015] A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the collaborative control method based on the Internet of Things and cloud computing.

[0016] Compared with the prior art, the collaborative control method, system and electronic device based on the Internet of Things and cloud computing provided by the present application use the Internet of Things and cloud computing technologies to determine the strength of the gripper based on the captured image of the fruit. In this way, the picking operation can be performed using the robot's autonomous collaborative operation mode, which can effectively prevent the picked fruits from breaking, and at the same time greatly reduce the labor cost of the picking operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 It is a flow chart of a collaborative control method based on the Internet of Things and cloud computing according to an embodiment of the present application.

[0019] Figure 2 The present invention is a flowchart of obtaining a fruit depth feature fusion map by passing the generated fruit region of interest through a depth feature fusion module in a collaborative control method based on the Internet of Things and cloud computing according to an embodiment of the present application.

[0020] Figure 3 It is a block diagram of a collaborative control system based on the Internet of Things and cloud computing according to an embodiment of the present application.

[0021] Figure 4 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Figure 1 FIG. 1 is a flow chart of a collaborative control method based on the Internet of Things and cloud computing according to an embodiment of the present application. Figure 1 As shown, according to the collaborative control method based on the Internet of Things and cloud computing according to the embodiment of the present application, it includes: S110, acquiring a captured image of the fruit; S120, performing feature extraction and encoding on the captured image of the fruit to obtain an optimized enhanced fruit feature vector; S130, passing the optimized enhanced fruit feature vector through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripper force should be increased or decreased.

[0024] In step S110, a photographic image of the fruit is obtained. As mentioned in the above background technology, with the development of artificial intelligence technology, picking robots have appeared in people's sight, which can greatly reduce labor costs. However, in the operation of picking robots, there are still cases where the fruit is rotten due to excessive or insufficient grasping force, or the fruit cannot be picked due to insufficient grasping force. Therefore, an optimized collaborative control method is expected.

[0025] In response to the above technical problems, a collaborative control method based on the Internet of Things and cloud computing is proposed, which uses the Internet of Things and cloud computing technologies to determine the strength of the gripper based on the captured image of the fruit. By acquiring the captured image of the fruit through the Internet of Things sensor and combining the image recognition and processing capabilities of cloud computing, the position, maturity and characteristics of the fruit can be intelligently determined, thereby providing a basis for the robot to make picking decisions. Based on the results of cloud computing analysis, the robot can autonomously adjust the strength and posture of the gripper to adapt to fruits of different shapes and sizes, thereby effectively preventing the fruit from being damaged during the picking process. In this way, using the robot's autonomous collaborative operation mode to perform picking operations can effectively prevent the picked fruit from breaking, while greatly reducing the labor cost of the picking operation.

[0026] At present, deep learning and neural networks have been widely used in computer vision, natural language processing, speech signal processing and other fields. In addition, deep learning and neural networks have also shown a level close to or even beyond that of humans in image classification, object detection, semantic segmentation, text translation and other fields.

[0027] In recent years, the development of deep learning and neural networks has provided new solutions and plans for collaborative control methods based on the Internet of Things and cloud computing.

[0028] Specifically, first, a photographic image of the fruit is obtained. By taking the image, the system can use image processing and computer vision technology to identify the location, size and maturity of the fruit. This is crucial for automated orchard picking operations, because the machine needs to know the exact location and attributes of the fruit to be able to pick it accurately.

[0029] In step S120, the captured image of the fruit is subjected to feature extraction and coding to obtain an optimized enhanced fruit feature vector. In order to improve the system's understanding and recognition capabilities of the fruit, the captured image of the fruit is subjected to feature extraction and coding to obtain an optimized enhanced fruit feature vector, thereby achieving a more accurate and intelligent grasping decision. Through feature extraction and coding, the system can extract important features related to the fruit from the image, such as information such as shape, size, color, texture, etc., which are crucial for the recognition and grasping decision of the fruit. At the same time, a large amount of irrelevant information may be contained in the original image. Through feature extraction and coding, the image information can be converted into a more representative and compact feature representation, which reduces the complexity of the data and improves processing efficiency. In this way, the optimized enhanced fruit feature vector obtained can better reflect the key features of the fruit. Through these feature vectors, the system can more accurately judge the maturity, shape, surface features, etc. of the fruit, providing a more reliable basis for the decision-making of grasping strength.

[0030] Specifically, in the above-mentioned collaborative control method based on the Internet of Things and cloud computing, the captured image of the fruit is feature extracted and encoded to obtain an optimized enhanced fruit feature vector, including: performing fruit target detection on the captured image of the fruit to generate a fruit region of interest; and, feature extraction and encoding of the generated fruit region of interest to obtain an optimized enhanced fruit feature vector.

[0031] Specifically, in order to automatically identify the fruit part in the image based on the acquired image and provide accurate location information for subsequent image processing and picking operations, the captured image of the fruit is subjected to fruit target detection to generate a fruit region of interest, thereby achieving a more intelligent and efficient orchard picking operation. Through the target detection network, the system can automatically identify the fruit position and boundary in the image, so that the specific position of the fruit can be accurately determined, thereby providing accurate target coordinates for subsequent processing and operations. The use of the target detection network can greatly improve the efficiency of the picking system. Compared with manual or traditional fixed rules, automated target detection can more quickly find and locate fruits, thereby reducing picking time and improving overall picking efficiency. At the same time, the fruit target detection network can adapt to fruits of different types and shapes, so it can also provide good adaptability to diverse orchard environments.

[0032] More specifically, in order to improve the image quality, the fruit region of interest is processed by an image clarity enhancer based on a generative adversarial network, making the subsequent steps more accurate and effective. The clarity enhancer can help improve the clarity and details of the image, making the fruit region of interest clearer and easier to identify. This is very important for the subsequent fruit recognition and positioning steps, because the clarity-enhanced image can provide more details and features, which helps to accurately identify the shape, color and maturity of the fruit. At the same time, the clarity-enhanced image can provide better input quality, which is conducive to subsequent image processing and analysis. For deep learning-based algorithms, clarity-enhanced images can provide a better foundation for subsequent processing.

[0033] Specifically, in the above-mentioned collaborative control method based on the Internet of Things and cloud computing, the captured image of the fruit is subjected to fruit target detection to obtain a fruit region of interest, including: passing the captured image of the fruit through a fruit target detection network to obtain a fruit region of interest; and passing the fruit region of interest through an image clarity enhancer based on a generative adversarial network to obtain a generated fruit region of interest.

[0034] Specifically, in the technical solution of the present application, the fruit target detection network is a target detection network based on anchor-free windows. The target detection network based on anchor-free windows is a target detection algorithm that does not rely on predefined anchor points or prior frames, but generates candidate frames densely on the image and uses a convolutional neural network to simultaneously predict the positions and target categories of these candidate frames. The target detection network based on anchor-free windows has some advantages in fruit target detection, which makes it an ideal choice for handling fruit detection tasks: 1. Strong adaptability: The target detection network based on anchor-free windows can better adapt to fruits of different sizes and proportions. Since the size and shape of fruits may vary greatly, the method using anchor-free windows can adapt to this diversity more flexibly without predefined anchor points. 2. Reduce design complexity: Compared with traditional anchor-based target detection methods, the anchor-free window method can reduce design complexity. In traditional methods, it is necessary to design and select anchor points suitable for different sizes and proportions, while the anchor-free window method achieves target detection by densely generating candidate frames in the image, reducing the need for this design. 3. Better adaptability: The anchor-free method is usually better able to adapt to the diversity of objects, including different shapes, sizes, and proportions, which is especially important for objects such as fruits that have large differences in appearance. 4. Reduced annotation costs: Compared with the anchor method, the anchor-free method can reduce the annotation cost. In the anchor method, positive and negative samples need to be annotated for each anchor point, while the anchor-free method can reduce the need for negative samples by generating dense candidate boxes, thereby reducing the annotation cost.

[0035] Specifically, the workflow of the target detection network based on anchor-free windows can be divided into the following steps: 1. Input image: The input of the network is the image of the target to be detected. 2. Feature extraction: The input image is feature extracted by a convolutional neural network (CNN). These features are usually extracted at multiple scales so that targets of different sizes can be detected. 3. Candidate box generation: Candidate boxes (also called anchor boxes) are densely generated on the feature map. These candidate boxes cover different areas of the image and have different sizes and aspect ratios to accommodate targets of different shapes and sizes. 4. Target classification and bounding box regression: Target classification and bounding box regression are performed on each candidate box. This step is usually implemented using convolutional layers and fully connected layers to predict the target category for each candidate box and adjust the bounding box position to more accurately frame the target location. 5. Non-maximum suppression (NMS): Non-maximum suppression is used to eliminate candidate boxes with high overlap to obtain the final target detection result. NMS retains the candidate boxes with the highest confidence and eliminates other candidate boxes that overlap more with these candidate boxes. 6. Output detection results: The output includes information such as target category, location, and confidence, as well as the final target detection result. These steps constitute the basic workflow of the anchor-free target detection network. In practical applications, different algorithms may have some subtle differences, but generally follow a similar process.

[0036] Specifically, in the above-mentioned collaborative control method based on the Internet of Things and cloud computing, the fruit target detection network is a target detection network based on a non-anchor window, and the target detection network based on a non-anchor window is YOLOv1, FCOS, CenterNet, ExtremeNet or RepPoints.

[0037] More specifically, by fusing the generated fruit regions of interest through the deep and shallow feature fusion module, the final fruit deep and shallow feature fusion map can have a richer, more accurate and more interpretable feature representation, providing a better basis for subsequent fruit analysis and decision-making. Deep features and shallow features are each good at capturing information at different levels. Deep features usually contain more abstract and semantic information, while shallow features focus more on local details and textures. By fusing deep and shallow features, the final fruit image can have richer and more comprehensive information, which is helpful for subsequent fruit analysis and understanding. Deep and shallow feature fusion can help improve the accuracy and robustness of fruit features. Fusion of features at different levels can make up for their respective shortcomings, thereby obtaining a more comprehensive and accurate representation of fruit features. This is very important for fruit recognition, classification and state assessment. Fusion of deep and shallow features can enhance the robustness of the image processing system to different lighting, angles and occlusions. Deep features may have a certain invariance to changes in lighting and rotation, while shallow features may be more adaptable to changes in texture and local features. Therefore, by fusing these features, fruit feature extraction can be made more robust.

[0038] Specifically, in the above-mentioned collaborative control method based on the Internet of Things and cloud computing, the generated fruit region of interest is feature extracted and encoded to obtain an optimized enhanced fruit feature vector, including: passing the generated fruit region of interest through a depth feature fusion module to obtain a fruit depth feature fusion map; passing the fruit depth feature fusion map through a convolutional neural network model including multiple mixed convolutional layers to obtain an enhanced fruit feature vector; and, performing entropy decay optimization on the enhanced fruit feature vector based on the classifier decision parameters to obtain an optimized enhanced fruit feature vector.

[0039] Figure 2 The flowchart of the method for collaborative control based on the Internet of Things and cloud computing according to the embodiment of the present application is to generate the fruit region of interest through the depth feature fusion module to obtain the fruit depth feature fusion map. Figure 2 As shown, the generated fruit region of interest is passed through a depth-shallow feature fusion module to obtain a fruit depth-shallow feature fusion map, including: S10, extracting a shallow feature map from the Mth layer of the depth-shallow feature fusion module, wherein M is greater than or equal to 1 and less than or equal to 6; S20, extracting a deep feature map from the Nth layer of the depth-shallow feature fusion module, wherein N / M is greater than or equal to 5 and less than or equal to 10; and S30, using the depth-shallow feature fusion module to fuse the shallow feature map and the deep feature map to obtain a fruit depth-shallow feature fusion map.

[0040] More specifically, by processing the fruit depth feature fusion map through a convolutional neural network model containing multiple mixed convolutional layers, an enhanced fruit feature vector can be obtained, so that the final fruit feature representation is richer, has better generalization ability, and is suitable for various fruit recognition and analysis tasks. Through the processing of the convolutional neural network (CNN) model, the feature expression can be further improved. Multiple mixed convolutional layers can enrich and enhance the information in the original feature map by learning complex feature representations, so that the final enhanced fruit feature vector is more discriminative and expressive. The mixed convolutional layer can extract abstract feature representations at different levels, from lower-level edges and textures to higher-level semantic features. This helps to capture various features in the fruit image, so that the final enhanced fruit feature vector has richer semantic information. The CNN model can recognize and capture various patterns and features in the image through training, which is very important for the recognition, classification and positioning of fruits. The enhanced fruit feature vector can help improve the performance of the fruit recognition system, so that the system can understand and process fruit images more accurately. At the same time, the CNN model can adaptively learn different types of fruit features, so it can adapt to fruit images of different types and states, making the enhanced fruit feature vector more widely applicable.

[0041] Specifically, in the above-mentioned collaborative control method based on the Internet of Things and cloud computing, the fruit depth feature fusion map is passed through a convolutional neural network model including multiple mixed convolutional layers to obtain a reinforced fruit feature vector, including: using each mixed convolutional layer of the convolutional neural network model to respectively perform input data in the forward transfer of the layer: using a first convolution kernel with a first size to perform convolution encoding on the fruit depth feature fusion map to obtain a first scale feature map; using a second convolution kernel with a first void rate to perform convolution encoding on the fruit depth feature fusion map to obtain a second scale feature map; using a third convolution kernel with a second void rate to perform convolution encoding on the fruit depth feature fusion map to obtain a third scale feature map; using a fourth convolution kernel with a third void rate to perform convolution encoding on the fruit depth feature fusion map The kernel performs convolution encoding on the fruit depth feature fusion map to obtain a fourth-scale feature map, wherein the first convolution kernel, the second convolution kernel, the third convolution kernel and the fourth convolution kernel have the same size, and the second convolution kernel, the third convolution kernel and the fourth convolution kernel have different void rates; the first-scale feature map, the second-scale feature map, the third-scale feature map and the fourth-scale feature map are aggregated along the channel dimension to obtain an aggregated feature map; and, the aggregated feature map is pooled to generate a pooled feature map; the pooled feature map is activated to generate an activated feature map; wherein the output of the last layer of the convolutional neural network model including multiple mixed convolution layers is the enhanced fruit feature vector.

[0042] In particular, in the technical solution of the present application, although the deep and shallow feature fusion module can extract low-level specific details and high-level semantic information, so as to better understand the image content, it may also produce overlapping features, because there may be certain similarities or correlations between deep features and shallow features, especially when processing similar types of features (such as textures, edges, etc.), which leads to feature redundancy in the final enhanced fruit feature vector. The existence of redundant features makes the feature space more crowded, causing the boundaries between different categories (such as fruits requiring different grasping forces) to become blurred. This makes it difficult for the classifier to accurately distinguish different situations, thereby affecting the classification performance. During the training process, redundant features will cause the model learning efficiency to decrease, because part of the computing resources and storage space are used to process useless information. In addition, redundant features may also cover up the truly useful features, causing the model to be difficult to converge to the optimal solution during the iterative optimization process. In order to solve this problem, in the technical solution of the present application, the enhanced fruit feature vector is optimized by performing entropy decay optimization based on the classifier decision parameters to obtain an optimized enhanced fruit feature vector.

[0043] Among them, the fortified fruit feature vector is optimized based on the entropy decay of the classifier decision parameters to obtain the optimized fortified fruit feature vector, including: extracting each row vector in the decision parameter space of the classifier to obtain a set of fortified fruit parameter row vectors; calculating the class probability entropy degradation simulation value between the fortified fruit feature vector and each fortified fruit parameter row vector in the set of fortified fruit parameter row vectors to obtain a set of class probability entropy degradation simulation values; calculating the square root of the sum of the squares of the class probability entropy degradation simulation values ​​to obtain an entropy degradation modulation value; based on the entropy degradation modulation value, performing feature distribution modulation on the fortified fruit feature vector to obtain the optimized fortified fruit feature vector.

[0044] The optimization formula is:

[0045] P = {p 1 ,p 2 ,...,p j ,...,p n}

[0046] D j =Softmax(M m v c +H m p j +b m )

[0047]

[0048]

[0049] Among them, P represents the set of row vectors of fruit strengthening parameters, p 1 、p 2 、p j 、p n represents the first, second, jth and nth fortified fruit parameter row vectors in the set of fortified fruit parameter row vectors, v c represents the enhanced fruit feature vector, Softmax represents the normalized exponential function, M m represents the first weight matrix, H m represents the second weight matrix, b m represents the bias vector, D j represents the jth class probability entropy degradation simulation value of the set of class probability entropy degradation simulation values, t represents the entropy degradation modulation value, v c ⊙t Indicates the calculation of the enhanced fruit feature vector v c Each eigenvalue of the t-th power, represents the i-th eigenvalue of the enhanced fruit feature vector, α represents a predetermined hyperparameter, represents addition by position, and V' represents the optimized enhanced fruit feature vector.

[0050] Specifically, in the technical solution of the present application, the feature vector of the enhanced fruit has feature redundancy, which causes inter-category crowding relative to the class label of the classifier when the classifier is used for classification judgment, thereby affecting the classification iteration effect.

[0051] Based on this, in the technical solution of the present application, the entropy attenuation optimization based on the classifier decision parameters is performed on the enhanced fruit feature vector, which first extracts each row vector from the prediction parameter space of the classifier to form a row vector set. Then, the class probability entropy degradation simulation value between the enhanced fruit feature vector and each row vector in the above row vector set is calculated to measure the uncertainty or information entropy change of the enhanced fruit feature vector under given parameters. Furthermore, all class probability entropy degradation simulation values ​​are squared and summed, and then the square root is taken to obtain a comprehensive entropy degradation modulation value, which reflects the degree of entropy change of the entire system when processing the feature vector. Finally, the enhanced fruit feature vector is adjusted according to the entropy degradation modulation value obtained above. Specifically, the attribute set of the enhanced fruit feature vector is represented in a power series based on the entropy degradation modulation value, and a set aggregate bounded deviation represented by an aggregation radius based on each attribute value of the enhanced fruit feature vector is added. Under a unified entropy degradation modulation value restriction standard, the enhanced fruit feature vector is subjected to attribute selection and sparse optimization to ensure that the enhanced fruit feature vector contains sufficient data but is not so cumbersome as to introduce unnecessary interference or redundancy.

[0052] In step S130, the optimized enhanced fruit feature vector is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripping force should be increased or decreased. Different fruits may require different gripping forces. The classification result can be used to personalize the gripping force according to the characteristics of the fruit, thereby improving the success rate and stability of fruit grasping. The classification result can provide the system with important information about the fruit, such as the maturity, position and surface features of the fruit, thereby helping the system make more accurate grasping decisions. The results obtained by the classifier can make the gripper system adaptive, and can dynamically adjust the gripping force according to the characteristics of different fruits, thereby being suitable for a variety of fruit grasping tasks.

[0053] Specifically, in the above-mentioned collaborative control method based on the Internet of Things and cloud computing, the classifier is used to process the optimized fortified fruit feature vector using the following classification formula to obtain the classification result;

[0054] Wherein, the classification formula is: softmax{(W n ,B n ):…:(W 1 ,B 1 )|V}, where W 1 To W n is the weight matrix, B 1 To B n is the bias vector, V is the optimized fortified fruit feature vector, softmax represents the normalized exponential function, and O represents the classification result.

[0055] In summary, in the technical solution of the present application, first, a captured image of the fruit is obtained. Then, the captured image of the fruit is passed through a fruit target detection network to obtain a fruit region of interest. Then, the fruit region of interest is passed through an image clarity enhancer based on a generative adversarial network to obtain a generated fruit region of interest. Then, the generated fruit region of interest is passed through a depth feature fusion module to obtain a fruit depth feature fusion map. Then, the fruit depth feature fusion map is passed through a convolutional neural network model including multiple mixed convolutional layers to obtain an enhanced fruit feature vector. Then, the enhanced fruit feature vector is optimized based on entropy decay of classifier decision parameters to obtain an optimized enhanced fruit feature vector. Furthermore, the optimized enhanced fruit feature vector is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripper's strength should be increased or decreased.

[0056] In summary, the collaborative control method based on the Internet of Things and cloud computing according to the embodiment of the present application has been explained, which uses the Internet of Things and cloud computing technology to determine the strength of the gripper based on the captured image of the fruit. In this way, using the robot's autonomous collaborative operation mode to perform the picking operation can effectively prevent the picked fruit from breaking, and at the same time greatly reduce the labor cost of the picking operation.

[0057] Based on the same inventive concept, the embodiments of the present application also provide a collaborative control system based on the Internet of Things and cloud computing, which can be used to implement the methods described in the above embodiments, as described in the following embodiments. Since the principle of solving the problem by the collaborative control system based on the Internet of Things and cloud computing is similar to the above method, the implementation of the collaborative control system based on the Internet of Things and cloud computing can refer to the implementation of the above method, and the repeated parts will not be repeated. As used below, the terms "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0058] Figure 3 FIG. 1 is a block diagram of a collaborative control system based on the Internet of Things and cloud computing according to an embodiment of the present application. Figure 3 As shown, according to the collaborative control system 100 based on the Internet of Things and cloud computing according to the embodiment of the present application, it includes: a fruit image acquisition module 110, used to acquire a captured image of the fruit; an optimization module 120, used to perform feature extraction and encoding on the captured image of the fruit to obtain an optimized enhanced fruit feature vector; a gripper force control module 130, used to pass the optimized enhanced fruit feature vector through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripper force should be increased or decreased.

[0059] Here, those skilled in the art can understand that the specific functions and operations of each unit and module in the collaborative control system 100 based on the Internet of Things and cloud computing have been described in the above reference. Figure 1 to Figure 2 The description of the collaborative control method based on the Internet of Things and cloud computing has been introduced in detail, and therefore, its repeated description will be omitted.

[0060] In summary, the collaborative control system based on the Internet of Things and cloud computing according to the embodiment of the present application has been explained, which uses the Internet of Things and cloud computing technologies to determine the strength of the gripper based on the captured image of the fruit. In this way, the picking operation can be performed using the robot's autonomous collaborative operation mode, which can effectively prevent the picked fruits from breaking, and at the same time greatly reduce the labor cost of the picking operation.

[0061] As described above, the collaborative control system 100 based on the Internet of Things and cloud computing according to the embodiment of the present application can be implemented in various terminal devices, such as a collaborative control server based on the Internet of Things and cloud computing. In one example, the collaborative control system 100 based on the Internet of Things and cloud computing according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the collaborative control system 100 based on the Internet of Things and cloud computing can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the collaborative control system 100 based on the Internet of Things and cloud computing can also be one of the many hardware modules of the terminal device.

[0062] Alternatively, in another example, the collaborative control system 100 based on the Internet of Things and cloud computing and the terminal device may also be separate devices, and the collaborative control system 100 based on the Internet of Things and cloud computing may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0063] In a typical example, the electronic device specifically includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the collaborative control method based on the Internet of Things and cloud computing are implemented.

[0064] Below, reference Figure 4 To describe an electronic device according to an embodiment of the present application. Figure 4 FIG. 1 is a block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device includes: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the collaborative control method based on the Internet of Things and cloud computing.

[0065] The processor may take the form of a computer-readable medium, logic gate, switch, application-specific integrated circuit (ASIC), programmable logic controller, and embedded microcontroller, etc., which stores computer-readable program codes (such as software or firmware) that can be executed by the (micro) processor. The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk (HDD), or memory card. The electronic device may be a server of the method described in the above embodiment, etc. Of course, the electronic device may also include module structures such as input devices and transmission modules. In this embodiment, the functions and effects specifically implemented by the electronic device can be explained in comparison with other embodiments, and will not be repeated here.

[0066] In addition, the embodiments of this specification also provide a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed, the steps of the method described in any one or more of the above embodiments are implemented. The storage medium may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The storage medium may include: devices that store information using electrical energy, such as various types of memories, such as RAM, ROM, etc.; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and U disks; devices that store information optically, such as CDs or DVDs. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, and so on.

[0067] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0068] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0069] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0070] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0071] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A collaborative control method based on the Internet of Things and cloud computing, characterized in that: include: Acquire a photographic image of the fruit; Extracting and encoding the captured image of the fruit to obtain an optimized enhanced fruit feature vector; The optimized fortified fruit feature vector is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripping force should be increased or decreased.

2. The collaborative control method based on the Internet of Things and cloud computing according to claim 1 is characterized in that: Extracting and encoding the captured image of the fruit to obtain an optimized enhanced fruit feature vector includes: Performing fruit target detection on the captured image of the fruit to generate a fruit region of interest; The generated fruit region of interest is subjected to feature extraction and encoding to obtain an optimized enhanced fruit feature vector.

3. The collaborative control method based on the Internet of Things and cloud computing according to claim 2 is characterized in that: Performing fruit target detection on the captured image of the fruit to generate a fruit region of interest includes: Passing the captured image of the fruit through a fruit object detection network to obtain a fruit region of interest; The fruit region of interest is passed through an image clarity enhancer based on a generative adversarial network to obtain a generated fruit region of interest.

4. The collaborative control method based on the Internet of Things and cloud computing according to claim 3 is characterized in that: The fruit object detection network is an anchor-free window-based object detection network.

5. The collaborative control method based on the Internet of Things and cloud computing according to claim 4 is characterized in that: Extracting and encoding the generated fruit region of interest to obtain an optimized enhanced fruit feature vector includes: The generated fruit region of interest is passed through a depth feature fusion module to obtain a fruit depth feature fusion map; Passing the fruit depth feature fusion map through a convolutional neural network model comprising multiple mixed convolutional layers to obtain a reinforced fruit feature vector; The fortified fruit feature vector is optimized by entropy decay based on classifier decision parameters to obtain an optimized fortified fruit feature vector.

6. The collaborative control method based on the Internet of Things and cloud computing according to claim 5 is characterized in that: The generated fruit region of interest is passed through a depth feature fusion module to obtain a fruit depth feature fusion map, including: Extracting a shallow feature map from the Mth layer of the deep and shallow feature fusion module, where M is greater than or equal to 1 and less than or equal to 6; Extracting a deep feature map from the Nth layer of the deep and shallow feature fusion module, wherein N / M is greater than or equal to 5 and less than or equal to 10; The shallow-depth feature fusion module is used to fuse the shallow feature map and the deep feature map to obtain a fruit shallow-depth feature fusion map.

7. The collaborative control method based on the Internet of Things and cloud computing according to claim 6 is characterized in that: The fruit depth feature fusion map is passed through a convolutional neural network model including multiple mixed convolutional layers to obtain a reinforced fruit feature vector, including: Each mixed convolutional layer of the convolutional neural network model is used to perform the following operations on the input data in the forward pass of the layer: Using a first convolution kernel with a first size to perform convolution encoding on the fruit depth feature fusion map to obtain a first scale feature map; Using a second convolution kernel with a first void rate to perform convolution encoding on the fruit depth feature fusion map to obtain a second scale feature map; Using a third convolution kernel with a second void rate to perform convolution encoding on the fruit depth feature fusion map to obtain a third scale feature map; Using a fourth convolution kernel with a third hole rate to perform convolution encoding on the fruit depth feature fusion map to obtain a fourth scale feature map, wherein the first convolution kernel, the second convolution kernel, the third convolution kernel and the fourth convolution kernel have the same size, and the second convolution kernel, the third convolution kernel and the fourth convolution kernel have different hole rates; Aggregating the first scale feature map, the second scale feature map, the third scale feature map, and the fourth scale feature map along the channel dimension to obtain an aggregated feature map; Performing pooling processing on the aggregated feature map to generate a pooled feature map; Performing activation processing on the pooled feature map to generate an activation feature map; Among them, the output of the last layer of the convolutional neural network model including multiple mixed convolutional layers is the enhanced fruit feature vector.

8. The collaborative control method based on the Internet of Things and cloud computing according to claim 7 is characterized in that: The fortified fruit feature vector is subjected to entropy decay optimization based on a classifier decision parameter to obtain an optimized fortified fruit feature vector, comprising: Extracting each row vector in the decision parameter space of the classifier to obtain a set of enhanced fruit parameter row vectors; Calculating the class probability entropy degradation simulation value between the fortified fruit feature vector and each fortified fruit parameter row vector in the set of fortified fruit parameter row vectors to obtain a set of class probability entropy degradation simulation values; Calculating the square root of the sum of squares of the class probability entropy degradation simulation values ​​to obtain an entropy degradation modulation value; Based on the entropy degradation modulation value, the fortified fruit feature vector is subjected to feature distribution modulation to obtain the optimized fortified fruit feature vector.

9. A collaborative control system based on the Internet of Things and cloud computing, characterized in that: include: A fruit image acquisition module is used to acquire the photographed image of the fruit; An optimization module, used for extracting and encoding features of the captured image of the fruit to obtain an optimized enhanced fruit feature vector; The gripping force control module is used to pass the optimized fortified fruit feature vector through a classifier to obtain a classification result, and the classification result is used to indicate whether the gripping force should be increased or decreased.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the collaborative control method based on the Internet of Things and cloud computing as described in any one of claims 1-8.

Images