Intelligent inspection method and system for farms based on AR equipment

Abstract

The present invention provides an intelligent inspection method and system for farms based on AR equipment. The inspection method includes: S1, installing an inspection application on the AR device, and connecting the AR device to the server; S2, collecting images or videos of the farm production equipment through the AR device, and transmitting the images or videos to the server; S3, analyzing the images or videos based on the deep learning algorithm model, automatically identifying and analyzing the production equipment in the images or videos, detecting whether there are abnormal conditions in the production equipment, and generating corresponding inspection reports, and transmitting the results to the application and AR device for display to the inspection personnel. The intelligent inspection method and system for farms based on AR equipment provided by the present invention have the following advantages: high degree of automation, remote guidance and strong collaboration functions, in addition, strong adaptive learning ability, and can continuously improve the accuracy and efficiency of the system through the adaptive learning ability of the deep learning algorithm model.

Description

Technical Field

[0001] The present invention relates to an intelligent inspection method and system for a breeding farm based on AR equipment. Background Art

[0002] At present, in the industrial and aquaculture production process, the safe and stable operation of various equipment is crucial to ensure production efficiency and production safety. Therefore, regular inspection and maintenance of equipment is an indispensable task, and the traditional manual inspection method has become increasingly prominent with low efficiency, low accuracy and high cost, which has also promoted the development of research on intelligent inspection technology.

[0003] In recent years, with the development of AR technology, artificial intelligence technology, and sensor technology, deep learning, as an important artificial intelligence technology, has received widespread attention and has been applied to the field of intelligent inspection. With the help of deep learning technology, the intelligent inspection system can quickly and accurately inspect the equipment, and can also perform status analysis, fault diagnosis, and predictive analysis, etc. This can not only improve the inspection efficiency, but also reduce maintenance costs, improve production efficiency and equipment operation stability. However, there is no intelligent inspection method or system for farms based on AR equipment in the prior art. Summary of the invention

[0004] The present invention provides an intelligent inspection method and system for farms based on AR equipment, which can effectively solve the above problems.

[0005] The present invention is achieved in that:

[0006] The present invention provides an intelligent inspection method for a farm based on AR equipment, comprising the following steps:

[0007] S1, install the inspection application on the AR device and connect the AR device to the server;

[0008] S2, collect images or videos of farm production equipment through AR devices, and transmit the images or videos to the server;

[0009] S3 analyzes images or videos based on deep learning algorithm models, automatically identifies and analyzes the production equipment in the images or videos, detects whether there are any abnormalities in the production equipment, generates corresponding inspection reports, and transmits the results to the application and AR devices for display to the inspection personnel.

[0010] The present invention provides an intelligent inspection system for farms based on AR equipment, comprising:

[0011] An AR device, on which an inspection application is pre-installed, and the AR device is connected to a server, and the AR device is used to collect images or videos of production equipment in the farm and transmit the images or videos to the server;

[0012] The server is used to analyze images or videos based on deep learning algorithm models, automatically identify and analyze production equipment in images or videos, detect whether there are any abnormalities in the production equipment, generate corresponding inspection reports, and transmit the results to applications and AR devices for display to inspection personnel.

[0013] The beneficial effects of the present invention are as follows: the intelligent inspection method and system for farms based on AR equipment provided by the present invention have the following advantages: high degree of automation: through the deep learning algorithm model, automatic identification and analysis of equipment can be realized, manual intervention can be reduced, and the efficiency of inspection and maintenance can be improved; strong remote guidance and collaboration functions: remote guidance and collaborative inspection can be realized through AR technology, and the convenience and operability of equipment inspection and maintenance can be improved; strong adaptive learning ability: through the adaptive learning ability of the deep learning algorithm model, the accuracy and efficiency of the system are continuously improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 It is a flow chart of the intelligent inspection method of farms based on AR equipment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0017] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0018] Reference Figure 1 As shown, a method for intelligent inspection of farms based on AR equipment includes the following steps:

[0019] S1, install the inspection application on the AR device and connect the AR device to the server;

[0020] S2, collect images or videos of farm production equipment through AR devices, and transmit the images or videos to the server;

[0021] S3 analyzes images or videos based on deep learning algorithm models, automatically identifies and analyzes the production equipment in the images or videos, detects whether there are any abnormalities in the production equipment, generates corresponding inspection reports, and transmits the results to the application and AR devices for display to the inspection personnel.

[0022] In step S2, the farm production equipment mainly includes feeding equipment, environmental control equipment, automatic feeding system, manure treatment device, etc. In order to realize automatic control, the feeding equipment, environmental control equipment, automatic feeding system, and manure treatment device generally include a main control display device. If a device fails, the corresponding main control display device generally displays a fault code. Therefore, images or videos can be collected by the main control display device corresponding to the feeding equipment, environmental control equipment, automatic feeding system, and manure treatment device.

[0023] As a further improvement, for general environmental control equipment, such as heating, temperature control, ventilation, and air quality related equipment, its operating status can be further collected. For example, the operating status of the fan can be collected to determine whether it is operating normally; the temperature controller can collect the specific ambient temperature and compare it with the set ambient temperature; the humidity controller can collect the specific humidity and compare it with the set ambient humidity, etc., without limitation here.

[0024] In step S3, the step of analyzing the image or video based on the deep learning algorithm model, automatically identifying and analyzing the production equipment in the image or video, and detecting whether there is an abnormality in the production equipment specifically includes:

[0025] The image of the main control display device on the production equipment in the image or video is obtained, and it is determined whether the image of the main control display device has a fault code. If yes, the fault code is further obtained, and then the corresponding fault code is matched with the fault code provided by the manufacturer, and finally the fault type and solution corresponding to the fault code are retrieved. The fault type and solution corresponding to the fault code are obtained by pre-entering the server.

[0026] In other embodiments, the image or video analysis based on the deep learning algorithm model can obtain the temperature control device in the video. When obtaining the temperature control device, the specific ambient temperature can be collected and compared with the set ambient temperature to determine whether the temperature control device is abnormal.

[0027] In other embodiments, the deep learning algorithm model-based image or video analysis can obtain the humidity control device in the video. When obtaining the humidity control device, the specific ambient humidity can be collected and compared with the set ambient humidity to determine whether the humidity control device is abnormal.

[0028] In other embodiments, the deep learning algorithm model-based image or video analysis can obtain the heating control device in the video. When obtaining the heating control device, the specific ambient temperature can be collected and compared with the set ambient temperature to determine whether the heating control device is abnormal.

[0029] In other embodiments, the deep learning algorithm model-based image or video analysis can obtain the ventilation control equipment in the video. When obtaining the ventilation control equipment, the ventilation operating status can be collected and compared with the set ventilation equipment operating status to determine whether the ventilation control equipment is abnormal.

[0030] As a further improvement, after obtaining the fault type, the intelligent inspection method for a farm based on AR equipment may further include:

[0031] S31, locate and track the AR device to realize the position and posture recognition during the equipment inspection. For example, SLAM technology can be used to locate and track the AR device. In this way, the location and number of the faulty production equipment can be accurately reflected in the subsequent inspection report.

[0032] The form of the inspection report is not limited and can be selected according to actual needs.

[0033] In other embodiments, as a further improvement, the intelligent inspection method of the farm based on AR equipment may further include:

[0034] S4, by using AR devices to overlay the device status in the real scene, directly guide the maintenance operation by observing the device status information, and transmit the operation process to the back-end server in real time through the AR device for recording and analysis. It can be understood that by overlaying the device status in the real scene with AR devices and directly guiding the maintenance operation, the efficiency of equipment maintenance can be greatly improved.

[0035] In other embodiments, as a further improvement, the intelligent inspection method of the farm based on AR equipment may further include:

[0036] S5, collecting the real-time feces image of each pen in the farm in real time, and judging whether the real-time feces image is normal according to the image recognition model.

[0037] In step S5, the construction of the image recognition model includes the following steps:

[0038] S51, obtaining multiple feces images;

[0039] S52, labeling the stool graph according to "normal" and "innormal", and performing big data training;

[0040] S53, outputting a feces-based image recognition model.

[0041] In step S51, the feces pattern can be obtained by collecting specific feces patterns during the breeding process in various places. This is because the breeds, food and feeding habits of pigs raised in breeding places across the country are completely different. If a unified standard picture is used, it may lead to inaccurate recognition in the later stage.

[0042] As a further improvement, the acquisition of multiple feces images can be obtained through a depth camera. In one embodiment, a Kinect depth camera can be used, and the Kinect depth camera uses a 3D structured light depth sensor composed of an infrared transmitter, a color RGB camera, and an infrared CMOS camera. The infrared transmitter actively projects modulated near-infrared light, and the infrared light will be reflected when it shines on the object in the field of view. The infrared camera receives the reflected infrared light, uses TOF technology to measure the depth, calculates the time difference of the light, and obtains the depth image of the feces based on the depth of the object. The reason for using a depth camera to obtain it is that normal feces are loose and clumping structures, and their depth structure needs to be obtained through a depth camera; while the feces of sick pigs are thin or pasty (planar structure), and it is difficult to obtain their depth structure, so it can be quickly distinguished and trained, greatly improving the efficiency of training.

[0043] In step S32, the step of labeling the feces pattern according to "normal" and "innormal" is mainly performed manually. As a further improvement, in one embodiment, during the labeling process, loose stools or mushy stools are labeled as "innormal", loose lumps are labeled as "normal", and granular stools are labeled as "normal-to be confirmed". This is because loose stools or mushy stools represent a more serious condition of the pigs, while granular stools are relatively mild and may be caused by lack of water.

[0044] The feces graph is subjected to big data training in a big data processing unit. In this embodiment, the processing is performed using Halcon software. The advantage of Halcon software is that it supports 5 times the number of 2D and 3D image acquisition devices and provides higher bit depth image processing. In one embodiment, the pre-trained network in Halcon is called: pretrained_dl_classifier_compact.hdl. In this embodiment, a total of 323 images are collected from the above-mentioned images, including 210 "normal" images (including 33 "normal-to be confirmed") and 113 "innormal" images. The images classified as "normal", "innormal" and "normal-to be confirmed" are stored in the "normal" and "innormal" and "normal-to be confirmed" folders in the same directory according to the labeled categories. The corresponding data set can be directly read using read_dl_classifier_data_set to obtain the category labels described in the file and the image.

[0045] Experiments show that the feces-based image recognition model provided by the embodiment of the present invention can achieve an accuracy of more than 97% for "normal" and "innormal" pig feces, that is, loose or pasty and granular or agglomerated. However, the recognition accuracy of "innormal" and "normal-to-be confirmed", that is, granular or agglomerated, can only reach about 65%.

[0046] In other embodiments, as a further improvement, the intelligent inspection method of the farm based on AR equipment may further include:

[0047] S6, when the real-time stool image is "innormal", the inspector is warned through the VR device.

[0048] In other embodiments, as a further improvement, the intelligent inspection method of the farm based on AR equipment may further include:

[0049] S7, obtaining the proportion of "innormal" in the real-time feces images of the same column, and raising the alarm level when it is higher than a preset value. This is because when the proportion of "innormal" in the real-time feces images of the same column is higher than a preset value, it indicates that infection may occur, so an alarm level needs to be raised.

[0050] The intelligent inspection method for farms based on AR equipment provided by the present invention has the following advantages:

[0051] High degree of automation: Through deep learning algorithm models, automatic identification and analysis of equipment can be achieved, reducing manual intervention and improving the efficiency of inspection and maintenance;

[0052] High degree of visualization: Through AR technology, the equipment status and other information are superimposed and displayed in the real scene, realizing visualized inspection and maintenance operations;

[0053] Strong remote guidance and collaboration functions: AR technology is used to achieve remote guidance and collaborative inspection, improving the convenience and operability of equipment inspection and maintenance;

[0054] Strong adaptive learning ability: Through the adaptive learning ability of the deep learning algorithm model, the accuracy and efficiency of the system are continuously improved;

[0055] Cloud deployment and management: Deploy the system in the cloud to achieve unified inspection and maintenance management of equipment worldwide and improve management efficiency.

[0056] The embodiment of the present invention further provides an intelligent inspection system for farms based on AR equipment, which includes:

[0057] An AR device, on which an inspection application is pre-installed, and the AR device is connected to a server, and the AR device is used to collect images or videos of production equipment in the farm and transmit the images or videos to the server;

[0058] The server is used to analyze images or videos based on deep learning algorithm models, automatically identify and analyze production equipment in images or videos, detect whether there are any abnormalities in the production equipment, generate corresponding inspection reports, and transmit the results to applications and AR devices for display to inspection personnel.

[0059] The present invention further provides a computer-readable storage medium having a program file stored thereon, and when the program file is executed by a processor, the above method is executed.

[0060] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent inspection method for farms based on AR equipment. It is characterized in that The following steps are involved: S1, install the inspection application on the AR device and connect the AR device to the server; S2, collect images or videos of farm production equipment through AR devices, and transmit the images or videos to the server; S3, analyzes images or videos based on deep learning algorithm models, automatically identifies and analyzes production equipment in images or videos, detects whether there are abnormalities in production equipment, generates corresponding inspection reports, and transmits the results to applications and AR devices for inspection personnel to display; The steps of analyzing the image or video based on the deep learning algorithm model, automatically identifying and analyzing the production equipment in the image or video, and detecting whether there is an abnormality in the production equipment specifically include: Obtain an image of a main control display device on a production device in an image or video, determine whether the image of the main control display device has a fault code, and if so, further obtain the fault code, then match the corresponding fault code with the fault code provided by the manufacturer, and finally retrieve the fault type and solution corresponding to the fault code; Further including: S5, collecting the real-time feces image of each pen in the farm in real time, and judging whether the real-time feces image is normal according to the image recognition model, wherein the construction of the image recognition model includes the following steps: S51, using a Kinect depth camera to obtain multiple feces images and their depth structures; S52, labeling the stool images as "normal" and "innormal", and performing big data training; in the labeling process, labeling loose or mushy stools as "innormal", loose and clumping stools as "normal", and granular stools as "normal-to be confirmed"; S53, outputting a feces-based image recognition model; S6, when the real-time stool image is "innormal", an inspector is warned through a VR device; S7, obtaining the proportion of "innormal" in the real-time feces images of the same column, and raising the alarm level when it is higher than a preset value.

2. The intelligent inspection method for farms based on AR equipment as claimed in claim 1, It is characterized in that In step S2, the farm production equipment includes feeding equipment, environmental control equipment, automatic feeding system, and manure treatment equipment.

3. The intelligent inspection method for farms based on AR equipment as claimed in claim 1, It is characterized in that The step of collecting images or videos of farm production equipment by using an AR device includes: The images or videos of the main control display device of the farm production equipment are collected through AR equipment.

4. The intelligent inspection method for farms based on AR equipment as claimed in claim 1, It is characterized in that The step of analyzing the image or video based on the deep learning algorithm model, automatically identifying and analyzing the production equipment in the image or video, and detecting whether there is an abnormality in the production equipment specifically includes: The image or video analysis based on the deep learning algorithm model can obtain the temperature control device in the video. When obtaining the temperature control device, the specific ambient temperature is collected and compared with the set ambient temperature to determine whether the temperature control device is abnormal.

5. The intelligent inspection method for farms based on AR equipment as claimed in claim 1, It is characterized in that The step of analyzing the image or video based on the deep learning algorithm model, automatically identifying and analyzing the production equipment in the image or video, and detecting whether there is an abnormality in the production equipment specifically includes: The deep learning algorithm model is based on analyzing images or videos to obtain the humidity control device in the video. When obtaining the humidity control device, the specific ambient humidity is collected and compared with the set ambient humidity to determine whether the humidity control device is abnormal.

6. The intelligent inspection method for farms based on AR equipment as claimed in claim 1, It is characterized in that After the fault type is obtained, the intelligent inspection method for the farm based on AR equipment further includes: S31, positioning and tracking the AR device to achieve position and posture recognition during equipment inspection.

7. The intelligent inspection method for farms based on AR equipment as claimed in claim 1, It is characterized in that The intelligent inspection method for farms based on AR equipment further includes: S4, uses AR devices to overlay the device status in the real scene, directly guides maintenance operations by observing device status information, and transmits the operation process to the backend server in real time through AR devices for recording and analysis.

Images