Intelligent field patrol system and method based on AR glasses

Abstract

The invention discloses a multifunctional field patrol interaction system based on AR glasses, and the system is characterized in that the system comprises an AR glasses terminal carrying optical display module, a camera, an ambient light sensor, and a positioning module; the cloud server is used for providing data storage, a crop phenotype analysis algorithm and a voice interaction module and adopting multi-mode interaction of gesture control, voice instructions and eye movement tracking; and the data analysis module is used for processing the image and video data in real time and outputting crop growth indexes and abnormal early warning. According to the invention, clear display in an outdoor high-brightness scene is realized by combining a strong-light environment adaptive display technology with an ambient light sensor and an AR optical coating, multi-modal interaction integrates gesture, voice and eye movement control, manual operation steps are reduced, a deep learning model is optimized by a lightweight phenotypic analysis algorithm, the AR glasses computing power is adapted, and the display effect is improved. And local real-time analysis is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart agriculture and precision farming management, and in particular, to an intelligent field inspection system and method based on AR glasses. Background Art

[0002] Traditional field inspection technologies rely on manual visual inspection or digital tools on mobile phones for assistance, and have the following defects that seriously conflict with agricultural timeliness: Information lag: Mobile phone photo records need to be uploaded and analyzed later, and sudden situations such as pests, diseases, drought, and waterlogging stress cannot be identified in real time, easily missing the best intervention opportunity; There is a delay in synchronizing manually recorded data with the cloud, and the latest field status cannot be shared during team collaborative operations. Operational inefficiency: The visibility of the mobile phone screen is poor under strong light, prolonging the time for field positioning and data entry, resulting in a limited area that can be inspected per day; The complex interaction of multi-level menu switching increases the probability of misoperation and affects the response speed to urgent farming operations. Decision-making isolation: Existing technologies lack the ability of multi-terminal real-time collaboration, and information such as abnormal areas marked by agronomists and pesticide application suggestions cannot be immediately synchronized to the executors, resulting in a disconnect between decision-making and execution. Irreversible risks in agricultural production: Environmental fluctuations or management mistakes at key nodes in the crop growth cycle will directly lead to reduced yields or even crop failures. Existing technologies are difficult to meet the closed-loop requirements of "real-time monitoring - instant analysis - synchronous execution", while the present invention compresses the timeliness fault tolerance window of farming operations from "hour level" to "minute level" through the collaborative architecture of AR glasses and the cloud, significantly reducing agricultural risks.

[0003] Therefore, there is an urgent need for a technology to change the existing field inspection technology and solve the problems of poor visibility under strong light, complex interaction, single function, and low collaborative efficiency in the existing field inspection technology. Summary of the Invention

[0004] The embodiments of the present application provide an intelligent field inspection system and method based on AR glasses, aiming to solve the problems of poor visibility under strong light, complex interaction, single function, and low collaborative efficiency in the existing field inspection technology.

[0005] Among them, according to one aspect of the embodiments of the present application, an intelligent field inspection system based on AR glasses is provided, including: an AR glasses terminal, a cloud server, a voice interaction module, and a data analysis module;

[0006] The AR glasses terminal is equipped with an optical display module, a camera, an ambient light sensor, and a positioning module;

[0007] The cloud server is used to provide data storage and crop phenotype analysis algorithms, and the analysis parameters of the data storage and crop phenotype analysis algorithms include plant height, number of leaves, and flower / fruit number recognition models;

[0008] The voice interaction module adopts multi-modal interaction of gesture control, voice commands, and eye movement tracking;

[0009] The data analysis module is used to process image and video data in real time and output crop growth indicators and anomaly warnings.

[0010] Furthermore, the AR eye terminal includes a 3D light column navigation unit, a strong light adaptive display unit, an expansion interface unit, a geographic information annotation unit, and a geographic information overlay unit.

[0011] Furthermore, the 3D light column navigation unit projects virtual light columns through the AR glasses to mark the field inspection path and the target area.

[0012] Furthermore, the strong light adaptive display unit dynamically adjusts the brightness of the AR interface through the ambient light sensor.

[0013] Furthermore, the voice interaction module includes a multimodal data acquisition unit. The multimodal data acquisition unit automatically associates data with geographical locations and timestamps by triggering one-key photo taking, video recording, and voice annotation.

[0014] Furthermore, the expansion interface unit is provided with reserved sensor interfaces for connecting environmental monitoring devices.

[0015] Furthermore, the environmental detection devices to be connected include devices such as temperature and humidity, and soil moisture.

[0016] Furthermore, the geographic information annotation unit annotates geographic information through the AR glasses. The geographic information includes irrigation situation survey information, pest and disease survey information, and grass situation survey information.

[0017] Furthermore, the geographic information overlay unit retrieves all historical information within the plot through the AR glasses. The historical information includes irrigation and pest and disease information.

[0018] Furthermore, the data analysis module includes a real-time phenotype analysis unit. The real-time phenotype analysis unit calculates crop parameters in real time based on an image recognition algorithm. The crop parameters include plant height, number of leaves, and stem diameter. When there is an abnormal area, an alarm is triggered.

[0019] Compared with the prior art, the beneficial effects of this application are as follows:

[0020] 1. The strong light environment adaptive display technology of this application combines the ambient light sensor and the AR optical coating to achieve clear display in outdoor high-brightness scenarios;

[0021] 2. The multimodal interaction of this application integrates gesture, voice, and eye movement control, reducing the number of manual operation steps;

[0022] 3. The lightweight phenotype analysis algorithm of this application optimizes the deep learning model, adapts to the computing power of AR glasses, and realizes local real-time analysis;

[0023] 4. The extensible architecture of this application supports plug-in access to third-party sensors and analysis modules (such as pest identification).

[0024] According to one aspect of the embodiments of this application, an intelligent field inspection method based on AR glasses is provided, including the following steps:

[0025] S100: The user loads the field inspection task through the AR glasses, and the system generates a 3D light column navigation path;

[0026] S200: After reaching the target area, the AR interface overlays the standard parameters of crop growth;

[0027] S300: Trigger taking pictures / videos through voice or gestures, and the system analyzes the image data in real time and feedbacks anomalies;

[0028] S400: The user enters remarks by voice, and the data is automatically uploaded to the cloud to generate an inspection report.

[0029] It can be understood that the above-mentioned intelligent field inspection system and method based on AR glasses have the same beneficial effects, which will not be elaborated here.

[0030] Next, through the drawings and embodiments, the technical solutions of this application will be further described in detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the specification depict the embodiments of this application and, together with the description, are used to explain the principles of this application.

[0032] Referring to the drawings, this application can be more clearly understood according to the following detailed description, where:

[0033] Figure 1 is the system architecture diagram of the intelligent field inspection system based on AR glasses proposed by this application;

[0034] Figure 2 is the schematic diagram of the AR glasses interface proposed by this application;

[0035] Figure 3 is the field inspection work flow chart proposed by this application;

[0036] Figure 4 is the flow chart of the intelligent field inspection system based on AR glasses proposed by this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present application.

[0038] Meanwhile, it should be understood that, for the sake of convenience of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationships.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended as any limitation on the present application or its application or use.

[0040] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the said technologies, methods, and devices should be regarded as part of the specification.

[0041] It should be noted that: like reference numerals and letters denote like items in the following drawings, and thus, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0042] In addition, the technical solutions between various embodiments of the present application may be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0043] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present application are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly.

[0044] The following will be combined with Figures 1 - 4 to describe an intelligent field inspection system and method based on AR glasses according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are only shown for the convenience of understanding the spirit and principle of the present application, and the embodiments of the present application are not limited in this regard. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0045] According to one aspect of the embodiments of the present application, an intelligent field inspection system based on AR glasses is provided, including: an AR glasses terminal, a cloud server, a voice interaction module, and a data analysis module;

[0046] The AR glasses terminal is equipped with an optical display module, a camera, an ambient light sensor, and a positioning module;

[0047] A cloud server for providing data storage and crop phenotyping analysis algorithms. The analysis parameters for data storage and crop phenotyping analysis algorithms include plant height, number of leaves, and recognition models for the number of flowers / fruits.

[0048] A voice interaction module that uses multi-modal interaction of gesture control, voice commands, and eye movement tracking.

[0049] A data analysis module for real-time processing of image and video data, and outputting crop growth indicators and anomaly warnings.

[0050] As can be seen from the above, in the data collection stage:

[0051] AR glasses terminal: The field inspection personnel wear AR glasses and enter the farmland. The camera equipped on the AR glasses terminal starts to capture images and videos of the farmland crops in real time. The ambient light sensor monitors the ambient light intensity in real time, providing environmental parameter references for subsequent image analysis. The positioning module obtains the location information of the farmland where the field inspection personnel are located, accurate to specific plots or even the positions of crop rows and plants.

[0052] Multi-modal interaction operation: During the field inspection process, the field inspection personnel can interact with the AR glasses terminal through the voice interaction module using multi-modal interaction methods such as gesture control, voice commands, or eye movement tracking. For example, when it is suspected that there is an anomaly in a certain area of the crops, the field inspection personnel can circle the crop area with a gesture or directly issue a voice command "Take pictures of these crops", and the AR glasses terminal will quickly respond and accurately collect relevant image and video data.

[0053] Data transmission and analysis stage:

[0054] Data transmission: The AR glasses terminal transmits the captured image, video data, as well as ambient light sensor data, positioning data, etc. to the cloud server in real time through a wireless network.

[0055] Cloud server analysis: After receiving the data, the cloud server processes the data using the pre-stored crop phenotyping analysis algorithm. This algorithm analyzes the crop phenotype based on parameters such as plant height, number of leaves, and recognition models for the number of flowers / fruits. For example, it measures the plant height through image recognition technology, counts the number of leaves, and identifies the number and growth status of flowers and fruits.

[0056] Data analysis module processing: The data analysis module obtains the analyzed results from the cloud server and further processes these image and video data in real time. Combining with the crop growth model and historical data, it calculates crop growth indicators such as crop growth rate and health index. At the same time, by comparing with the preset normal growth range, it determines whether there are any anomalies in the crops. If it is found that the crop growth indicators exceed the normal range, the data analysis module immediately generates anomaly warning information.

[0057] Results presentation and feedback stage:

[0058] AR glasses terminal presentation: The cloud server transmits the analysis results including crop growth indicators and abnormal warning information back to the AR glasses terminal. The AR glasses terminal uses an optical display module to intuitively present this information to the field patrol personnel in an augmented reality way. For example, in the field of view of AR glasses, for a certain crop, its height, number of leaves and other growth indicators are directly displayed, and abnormal warning information such as "yellowing leaves may indicate nitrogen deficiency" is displayed in eye-catching colors and text. Field patrol personnel can take corresponding measures in a timely manner, such as adjusting fertilization plans, arranging pest and disease control, etc., to achieve precise management of crop growth in farmland.

[0059] It can be understood that multi-terminal collaborative interaction supports users to directly mark patrol areas through computers or AR glasses. The marked information is synchronized to the cloud in real time and pushed to all online AR terminals to realize multi-user collaborative work; dynamic interaction enhancement integrates the multimodal interaction (voice, gesture, eye movement) of AR glasses and real-time synchronization technology on the cloud to simplify data sharing and task allocation processes in team collaboration; closed-loop data links interact with the entire process from labeling, analysis to feedback, avoiding information islands and operation delays in traditional patrols, and improving abnormal response speed and decision consistency.

[0060] Specifically, the AR eye terminal includes a 3D light column navigation unit, a strong light adaptive display unit, a multi-mode expansion interface unit, a geographic information marking unit and a geographic information overlay unit.

[0061] Specifically, the 3D light beam navigation unit projects virtual light beams through AR glasses to mark the patrol path and target area.

[0062] Specifically, the strong light adaptive display unit dynamically adjusts the brightness of the AR interface through the ambient light sensor.

[0063] It is understandable that when there are target areas in the farmland that need special attention, the 3D light beam navigation unit can accurately mark the location of these target areas through virtual light beams. Field patrollers wearing AR glasses can easily identify and quickly reach the target area even at a long distance. This precise positioning capability greatly saves time in finding the target area and improves the pertinence and efficiency of field patrol work.

[0064] Specifically, the voice interaction module includes a multimodal data acquisition unit, which uses a one-key triggering method for taking photos, video recording and voice annotation to automatically associate the data with a geographic location and a timestamp.

[0065] It is understandable that the collected data is automatically associated with the geographical location, accurate to the specific longitude and latitude information. This gives each piece of data a clear spatial location, facilitating spatial analysis and comparison in subsequent analysis. For example, through the map display function, the differences in crop growth conditions in different regions can be visually seen, and the spread path and distribution pattern of pests and diseases in the farmland can be analyzed. At the same time, when it is necessary to recheck a specific area or take targeted measures, the specific location can be quickly located according to the geographical location, improving the accuracy and efficiency of the work.

[0066] Specifically, the extended interface unit is provided with reserved sensor interfaces for connecting environmental monitoring devices.

[0067] Specifically, the connected environmental detection devices include temperature and humidity, soil moisture and other devices.

[0068] It is understandable that combining data such as temperature and humidity, and soil moisture with the crop's own growth indicators (such as plant height, number of leaves, etc.) can provide a more comprehensive environmental portrait for crop growth. For example, researchers can analyze the growth rate and nutrient absorption of crops under different soil moisture and temperature and humidity conditions, so as to more deeply understand the crop growth law and provide a basis for formulating a more scientific planting plan.

[0069] Specifically, the geographical information annotation unit annotates geographical information through AR glasses. The geographical information includes irrigation situation investigation information, pest and disease investigation information, and grass situation investigation information.

[0070] Specifically, the geographical information overlay unit retrieves all historical information within the plot through AR glasses. The historical information includes irrigation, pest and disease information.

[0071] It is understandable that the geographical information annotation unit provides a convenient way to collect various information about the farmland. The information annotated by different field inspection personnel can be automatically integrated into a unified database to form a comprehensive and systematic farmland information file. For example, the pest and disease annotation information of the same piece of farmland by multiple field inspection personnel at different times can be summarized in the database to reflect the development and change process of pests and diseases.

[0072] The historical information presented by the geographical information overlay unit is the accumulation of agricultural production practice experience. This information can be passed on among agricultural practitioners. Newly recruited field inspection personnel or young farmers can quickly learn and understand the common problems and treatment methods of this plot and similar farmlands by viewing the historical information, accelerating the inheritance of knowledge.

[0073] Specifically, the data analysis module includes a real-time phenotype analysis unit. Based on an image recognition algorithm, the real-time phenotype analysis unit calculates crop parameters in real time. The crop parameters include plant height, number of leaves, and stem diameter. When an abnormal area is detected, an alarm is triggered.

[0074] As can be seen from the above, in the application of the image recognition algorithm: after the cloud server receives the data, the real-time phenotype analysis unit uses the image recognition algorithm to process crop images and videos. Common image recognition algorithms include convolutional neural networks (CNNs), etc. These algorithms can automatically identify crop features in images and perform quantitative analysis.

[0075] Calculation of plant height: By analyzing the crop image, the top and bottom positions of the crop are identified. Using the scale or known reference object in the image, the actual plant height of the crop is calculated. For example, a ruler with a known length is preset in the image. By comparing the height of the crop in the image with the length ratio of the ruler, the plant height of the crop is calculated.

[0076] Counting the number of leaves: Using the image recognition algorithm to segment and identify crop leaves, and counting the number of leaves. The algorithm can distinguish according to features such as the color, shape, and texture of the leaves to improve the accuracy of counting.

[0077] Measurement of stem diameter: Locate the position of the crop stem in the image. By measuring the width of the stem in the image and combining with the scale, the actual thickness of the stem is calculated. To improve the measurement accuracy, multiple positions of the stem can be measured and the average value is taken as the final result.

[0078] Data comparison and analysis: The real-time phenotype analysis unit compares and analyzes the calculated crop parameters with the preset normal range. The normal range can be set according to factors such as crop variety, growth stage, and planting environment. For example, for a certain variety of wheat, the normal range of plant height during the jointing stage is 30 - 50 cm, the number of leaves is 5 - 7, and the stem diameter is 0.3 - 0.5 cm. When the calculated crop parameters exceed the normal range, the area is determined as an abnormal area.

[0079] It can be understood that the alarm rule is set according to actual needs. When the plant height of the crop is less than 20% of the normal range, the number of leaves decreases by more than 30%, or the stem diameter is less than 50% of the normal range, an alarm is triggered. The alarm rule can be adjusted according to different crop varieties and growth stages to improve the accuracy and pertinence of the alarm.

[0080] Alarm method: When an abnormal area is detected, the real-time phenotype analysis unit triggers an alarm. The alarm information can be notified to the field patrol personnel in various ways, such as displaying prominent alarm prompt information on the optical display module of the AR glasses and simultaneously emitting a sound alarm; or sending the alarm information to the mobile phone of the field patrol personnel via text message, push notification, etc.

[0081] Compared with the prior art, the beneficial effects of the present application are as follows: The strong light environment adaptive display technology of the present application combines an ambient light sensor and an AR optical coating to achieve clear display in outdoor high-brightness scenarios; the multimodal interaction integrates gesture, voice, and eye movement control to reduce manual operation steps; the lightweight phenotypic analysis algorithm optimizes the deep learning model to adapt to the computing power of the AR glasses and achieve local real-time analysis; the scalable architecture supports plug-in access to third-party sensors and analysis modules (such as pest and disease identification).

[0082] According to one aspect of the embodiments of the present application, a smart field patrol method based on AR glasses is provided. It can be understood that the above-mentioned smart field patrol system and method based on AR glasses have the same beneficial effects and will not be elaborated here.

[0083] Embodiment 1:

[0084] The user wears the AR glasses to start the field patrol task. The system plans the path through GPS / Beidou positioning, and the AR interface projects a green light column to guide the direction.

[0085] After reaching the target area, the camera captures the crop image. The local model calculates the plant height in real time and compares it with the preset threshold. If it is abnormal, the AR interface marks a red frame and gives a voice reminder.

[0086] The user swipes with a gesture to bring up the menu, selects "Record", and then the system saves a 10-second video clip and automatically associates the location information with the cloud database.

[0087] Embodiment 2:

[0088] Connect an infrared camera and a multispectral sensor to expand the system to support leaf disease identification.

[0089] After detecting the disease, the AR interface superimposes a recommended pesticide application plan and generates a purchase list and pushes it to the user's mobile phone.

[0090] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0091] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. An intelligent field inspection system based on AR glasses, characterized in that: include: AR glasses terminal, cloud server, voice interaction module and data analysis module; The AR glasses terminal is equipped with an optical display module, a camera, an ambient light sensor and a positioning module; The cloud server is used to provide data storage and crop phenotyping analysis algorithm, wherein the data storage and crop phenotyping analysis algorithm analysis parameters include plant height, leaf number, and flower / fruit number recognition model; The voice interaction module adopts a multi-modal interaction mode of gesture control, voice command, and eye tracking; The data analysis module is used to process image and video data in real time and output crop growth indicators and abnormal warnings.

2. The intelligent field inspection system based on AR glasses according to claim 1 is characterized in that: The AR glasses terminal includes a 3D light column navigation unit, a strong light adaptive display unit, an extended interface unit, a geographic information marking unit and a geographic information superposition unit.

3. The intelligent field inspection system based on AR glasses according to claim 2 is characterized in that: The 3D light column navigation unit projects a virtual light column through AR glasses to mark the patrol path and target area.

4. The intelligent field inspection system based on AR glasses according to claim 2 is characterized in that: The strong light adaptive display unit dynamically adjusts the brightness of the AR interface through the ambient light sensor.

5. The intelligent field inspection system based on AR glasses according to claim 2 is characterized in that: The extended interface unit is provided with a reserved sensor interface for accessing the environmental monitoring device.

6. The intelligent field inspection system based on AR glasses according to claim 2 is characterized in that: The geographic information marking unit marks geographic information through AR glasses, and the geographic information includes irrigation situation survey information, pest and disease survey information, and grass condition survey information.

7. The intelligent field inspection system based on AR glasses according to claim 2 is characterized in that: The geographic information overlay unit retrieves all historical information of the plot through AR glasses, and the historical information includes irrigation and pest information.

8. The intelligent field inspection system based on AR glasses according to claim 1 is characterized in that: The voice interaction module includes a multimodal data acquisition unit, which automatically associates the data with a geographic location and a timestamp by using a one-key triggering method for taking photos, video recording, and voice annotation.

9. The intelligent field inspection system based on AR glasses according to claim 1 is characterized in that: The data analysis module includes a real-time phenotyping analysis unit, which calculates crop parameters in real time based on an image recognition algorithm. The crop parameters include plant height, number of leaves, and stem thickness, and triggers an alarm when an abnormal area exists.

10. An intelligent field inspection method based on AR glasses, characterized in that: The following steps are involved: S100: The user loads the field patrol task through AR glasses, and the system generates a 3D light beam navigation path; S200: After reaching the target area, the AR interface overlays the standard parameters of crop growth; S300: Photo / video recording is triggered by voice or gesture, and the system analyzes image data in real time and reports abnormalities; S400: Users can input notes by voice, and the data will be automatically uploaded to the cloud to generate inspection reports.

Images