Smart farm inspection live broadcast system

Abstract

The invention discloses an intelligent farm inspection live broadcast system, which comprises a field subsystem, a service subsystem and a terminal subsystem, and is characterized in that the field subsystem is provided with a mobile inspection module which is used for farm inspection and shooting multi-view videos; and the service subsystem comprises a front-end edge computing module and a service center module. The former is responsible for splicing inspection videos into a panorama and then coding; and the crop identification module constructs a crop identification model, processes the coded data and outputs an identification result. And the terminal subsystem comprises a user terminal and a user mode control terminal. The user terminal supports live video broadcasting, playback, farm state pushing and interaction instruction sending; and the user mode control terminal is used for switching the control mode of the mobile inspection module. According to the technical scheme, the accuracy and efficiency of inspection can be improved, a brand new view angle and an interaction mode are brought to agricultural management, and intelligent development of agricultural production is promoted.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart agriculture, and in particular relates to a smart farm inspection and live broadcast system. Background Art

[0002] With the development of modern agriculture and the advancement of science and technology, the concept of smart farms has gradually emerged. Smart shared farms are a relatively new agricultural model that has emerged in China in recent years. Through the intelligence and informatization of agricultural production, it improves agricultural production efficiency, agricultural product quality, and interactive fun. Smart farms combine advanced technologies such as the Internet of Things, cloud computing, and big data to improve agricultural production efficiency, reduce labor costs, and increase crop yields and quality. The traditional farm inspection model mainly relies on manual labor, which is time-consuming and labor-intensive, and is prone to missed inspections and wrong inspections, making it difficult to meet the efficient management needs of modern agriculture. Summary of the invention

[0003] The purpose of the present invention is to provide a smart farm inspection and live broadcast system to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above-mentioned purpose, the present invention provides a smart farm inspection live broadcast system, including a field subsystem, a service subsystem and a terminal subsystem;

[0005] The field subsystem includes:

[0006] Mobile inspection module, used to conduct mobile inspections within the farm and capture multi-angle inspection video data during the inspection process;

[0007] The service subsystem includes:

[0008] The front-end edge computing module is used to perform panoramic stitching on the inspection video data to obtain a panoramic video; and encode the panoramic video to obtain encoded data;

[0009] A business center module is used to construct a crop identification model, and execute the crop identification task of the coded data based on the crop identification model to obtain a crop identification result;

[0010] The terminal subsystem includes:

[0011] User terminal, used for live video streaming, video streaming playback, farm status push and user interaction command sending;

[0012] The user mode control terminal is used to switch the user's control method for the mobile inspection module.

[0013] Optionally, the mobile inspection module includes:

[0014] A navigation submodule, used to obtain environmental information and plan the mobile inspection path of the vehicle based on the environmental information;

[0015] The communication submodule is used to communicate with the service subsystem and transmit control instructions and video data;

[0016] The video acquisition submodule is used to capture multi-view inspection video data around the vehicle.

[0017] Optionally, the field subsystem further includes:

[0018] The fixed bird's-eye view camera module is used to capture bird's-eye view video data of the farm based on a preset position.

[0019] Optional, front-end edge computing module, including:

[0020] The panoramic stitching submodule is used to perform image registration, image fusion and image stitching processing on the inspection video data of each channel to obtain a panoramic video;

[0021] The encoding submodule is used to perform block processing on the panoramic video to obtain a plurality of rectangular blocks. In the same frame, the pixel values ​​of the adjacent rectangular blocks of the panoramic video frame are used to predict the pixel values ​​of the current rectangular block, and the pixel values ​​corresponding to each rectangular block are quantized, encoded and filtered to obtain encoded data.

[0022] Optionally, the business center module includes:

[0023] The crop recognition model building submodule is used to build a crop recognition model based on a small sample target detection model;

[0024] A model training submodule is used to obtain training data, wherein the training data includes training image data and corresponding crop labels, divide the training data into a proxy base class set and a proxy new class set, and construct an initial crop recognition model based on the proxy base class set; construct a non-overlapping first data set, a second data set, and a third data set based on the proxy base class set and the proxy new class set; the first data set includes samples of the proxy base class set, which are used for model training for meta-adjustment, the second data set and the third data set both include samples of the proxy base class set and the proxy new class, the second data set is used to fine-tune the image during model meta-adjustment, and the third data set is used to evaluate the performance of the fine-tuning model;

[0025] Training and evaluation are performed on multiple proxy tasks based on scenario learning, specifically including: sampling a support set in a first data set, simulating a new category of proxy tasks based on the support set, sampling a loss function and an enhancement amplitude parameter, and fine-tuning an initial crop recognition model on the support set based on a gradient optimization method. During the fine-tuning process, the parameters of the initial crop recognition model are updated according to the direction of gradient descent to minimize the loss function; after the fine-tuning is completed, an average accuracy score of the initial crop recognition model is calculated based on a third data set;

[0026] Repeat the scenario learning process to obtain several average accuracy scores, and use the reinforcement learning algorithm to optimize the model parameter configuration after each scenario learning;

[0027] The model is trained through the scenario learning process and the model parameter configuration optimization process to obtain a trained crop recognition model;

[0028] The intelligent identification submodule is used to input the encoded data into the trained crop identification model for identification prediction to obtain the corresponding crop identification results.

[0029] Optionally, the service subsystem further includes:

[0030] The artificial intelligence acceleration module is used to access the business center module and provide computing acceleration services for the crop identification algorithm.

[0031] Optionally, the user mode control terminal includes:

[0032] Exclusive module for manually controlling the mobile inspection route and camera viewing angle;

[0033] Shared module for virtually changing the camera perspective.

[0034] Optionally, the sharing module includes:

[0035] The dynamic cropping submodule is used to dynamically crop the panoramic video according to the user interaction data to simulate the change of viewing angle;

[0036] The video stream automatic switching submodule is used to monitor the power and maintenance requirements of the mobile inspection module in real time. When the mobile inspection module needs to be offline, it automatically switches to the video source of the fixed overhead camera module;

[0037] The dynamic resource allocation submodule is used to dynamically allocate mobile inspection resources according to the number of user terminals.

[0038] The technical effects of the present invention are:

[0039] The present invention uses mobile carts, cameras and other equipment to collect dynamic data of the farm in real time, and broadcasts it live over the Internet, making it convenient for farmers and managers to monitor the farm status anytime and anywhere. In terms of video processing, the present invention uses video splicing technology, which can efficiently splice multiple captured video streams to generate panoramic videos. This panoramic video not only provides an intuitive visual experience for farm managers, but also can share the dynamic situation of the farm with relevant personnel in a timely manner through real-time live broadcast, facilitating quick decision-making. The smart farm inspection live broadcast system not only improves the accuracy and efficiency of inspections, but also brings a new perspective and interactive method to agricultural management, promoting the intelligent development of agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 is a general structural block diagram of an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of video splicing in an embodiment of the present invention;

[0044] Figure 3 is a flow chart of crop identification in an embodiment of the present invention;

[0045] Figure 4 This is a video stream automatic switching solution in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but should be understood as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0047] It should be understood that the terms described in the present invention are only for describing special embodiments and are not intended to limit the present invention. In addition, for the numerical range in the present invention, it should be understood that each intermediate value between the upper and lower limits of the scope is also specifically disclosed. Each smaller range between the intermediate value in any stated value or stated range and any other stated value or intermediate value in the described range is also included in the present invention. The upper and lower limits of these smaller ranges can be independently included or excluded in the scope.

[0048] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present invention description without departing from the scope or spirit of the present invention. Other embodiments derived from the present invention description will be apparent to those skilled in the art. The present application description and examples are exemplary only.

[0049] The words “include,” “including,” “have,” “contain,” etc. used in this article are open-ended terms, meaning including but not limited to.

[0050] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0051] Embodiment 1

[0052] like Figure 1 - Figure 4 As shown, this embodiment provides a smart farm inspection live broadcast system, including a field subsystem, a service subsystem and a terminal subsystem;

[0053] The field subsystem includes:

[0054] Mobile inspection module, used to conduct mobile inspections within the farm and capture multi-angle inspection video data during the inspection process;

[0055] The service subsystem includes:

[0056] The front-end edge computing module is used to perform panoramic stitching on the inspection video data to obtain a panoramic video; and encode the panoramic video to obtain encoded data;

[0057] A business center module is used to construct a crop identification model, and execute the crop identification task of the coded data based on the crop identification model to obtain a crop identification result;

[0058] The terminal subsystem includes:

[0059] User terminal, used for live video streaming, video streaming playback, farm status push and user interaction command sending;

[0060] The user mode control terminal is used to switch the user's control method for the mobile inspection module.

[0061] The system of this embodiment uses mobile carts, cameras and other equipment to collect dynamic data of the farm in real time and broadcast it live over the Internet, making it convenient for farmers and managers to monitor the farm status anytime and anywhere. The system integrates technologies such as target recognition, data analysis and remote control.

[0062] In terms of video processing, this embodiment uses video stitching technology, which can efficiently stitch multiple captured video streams to generate panoramic videos. This panoramic video not only provides an intuitive visual experience for farm managers, but also can share the dynamic situation of the farm with relevant personnel in a timely manner through real-time live broadcast, facilitating quick decision-making. The smart farm inspection live broadcast system not only improves the accuracy and efficiency of inspections, but also brings a new perspective and interactive method to agricultural management, promoting the intelligent development of agricultural production.

[0063] This embodiment proposes a smart farm inspection live broadcast system, including the following subsystems:

[0064] The on-site subsystem includes two components: a mobile inspection vehicle and a fixed overhead camera;

[0065] The service subsystem includes three components: front-end edge computing module, business center module and artificial intelligence acceleration module;

[0066] The terminal subsystem includes two components: the user terminal and the dedicated APP.

[0067] The mobile inspection vehicle in the field subsystem can move around the farm. It is equipped with a navigation module that can autonomously obtain environmental information and plan the moving path. The communication module can communicate with the service subsystem to transmit control instructions and video data. The video acquisition module is composed of multiple cameras at different angles, which can shoot multi-angle videos around the vehicle, including panoramic video information around the vehicle. The fixed bird's-eye view camera component in the field subsystem can be composed of one camera or an array of multiple cameras, providing a bird's-eye view of the farm taken from a high altitude.

[0068] The front-end edge computing module in the service subsystem is deployed near the farm, receiving multiple video streams from the field subsystem, summarizing the panoramic video information of the car, and compressing it into a panoramic video stream using a dedicated encoder. The business center module is deployed in the cloud server in the data center, providing intelligent recognition and computing of video streams, database services, live streaming services, two-way forwarding of instructions, and terminal subsystem access services. The artificial intelligence acceleration module is connected to the business center to provide computing acceleration services for the intelligent recognition algorithm.

[0069] The user terminals in the terminal subsystem mainly refer to smartphones and tablet devices running the Android system. The dedicated APP provides functions such as live video streaming and playback, farm status push, and user interaction command sending.

[0070] The front-end edge computing module contains a dedicated encoder for compressing multiple video streams from the car, which is characterized by: splicing multiple video streams into one video stream for unified processing, considering the spatial characteristics of panoramic video, for the leftmost image block of the video stream, its adjacent reference block includes the rightmost image block of the video stream, and for the rightmost image block of the video stream, its adjacent reference block includes the leftmost image block of the video stream. This encoding method can further improve the compression rate of the video stream and reduce the required network bandwidth and transmission delay.

[0071] The business center module contains an intelligent recognition algorithm suitable for small sample crops, which is characterized by:

[0072] The algorithm first uses a large number of common crops as training sets to obtain a basic model. In order to improve the recognition rate of specific crops grown on the farm, no more than 20 photos of the crop under different angles and lighting conditions can be taken as a support set, and the basic model can be further fine-tuned to obtain a small sample recognition model. The small sample recognition model can greatly improve the recognition accuracy of specific crops, and does not require photos of specific crops for large-scale training. It has the characteristics of high real-time performance, such as on-site shooting, on-site fine-tuning, and on-site recognition.

[0073] The dedicated APP of the terminal subsystem includes a video stream automatic switching solution, which is characterized by:

[0074] Users can control the mobile inspection cart in exclusive or shared mode. In exclusive mode, not only can the camera position be controlled to change the viewing angle, but the inspection path of the cart can also be manually controlled. In shared mode, users do not change the physical state of the cart, but can virtually change the camera viewing angle by dynamically cropping a portion of the panoramic video to simulate the effect of the change in viewing angle. If the cart is offline due to charging, maintenance, etc., the video stream automatically switches to a fixed overhead camera to avoid a black screen. When the number of user terminals is greater than the number of mobile inspection carts, this automatic switching solution can dynamically allocate cart resources to meet the needs of multiple users to concurrently view the farm status.

[0075] The implementation process of this embodiment includes:

[0076] Step 1: The mobile inspection car in the field subsystem takes multi-view videos of the surrounding area, and stitches the multi-view videos into a panoramic video using video stitching technology. Figure 2 shown.

[0077] Step 2: Further process the panoramic video stitched in step 1 and encode it using a dedicated encoder, as follows:

[0078] (1) Block division: The input video frame is divided into multiple rectangular blocks. This block division can improve the coding efficiency because the pixel values ​​of adjacent blocks usually have strong correlation and the redundant information can be reduced through prediction and transformation.

[0079] (2) Prediction: Use the pixel values ​​of adjacent blocks to predict the pixel values ​​of the current block. Prediction can be divided into intra-frame prediction and inter-frame prediction. Intra-frame prediction uses the pixel values ​​of adjacent blocks in the same frame for prediction, while inter-frame prediction uses the pixel values ​​of adjacent frames for prediction. For intra-frame prediction, in a circular layout, for the leftmost image block, its adjacent reference blocks include not only its direct adjacent blocks in the video stream, but also the rightmost image block in the video stream. The purpose of this is to maintain the spatial continuity of the panoramic video and ensure that when the user switches from the leftmost perspective to the rightmost perspective during video playback, the picture can transition seamlessly without any breaks or jumps. Transformation: Convert the prediction residual to the frequency domain for subsequent quantization and encoding. Similarly, for the rightmost image block in the video stream, its adjacent reference blocks should also include the leftmost image block in the video stream.

[0080] (3) Quantization: Quantize the transformed data to reduce data redundancy. The quantization process divides the transformed data by a quantization step and then rounds it to the nearest integer. The size of the quantization step affects the encoding quality and compression rate. The larger the quantization step, the worse the encoding quality, but the higher the compression rate.

[0081] (4) Encoding: Encode the quantized data to generate a compressed bit stream.

[0082] (5) Loop filtering: Filter the encoded bit stream to reduce coding distortion such as blocking effects and mosquito noise.

[0083] Through the above operations, video coding technology can achieve high compression and reconstruction of video signals, thereby ensuring video quality while achieving effective utilization of bandwidth and storage resources.

[0084] Step 3: A small sample-based crop recognition algorithm is integrated into the business center module, which only requires a few target samples to accurately identify the target crops. This embodiment follows the general small sample target detection (FS OD) setting, in which a relatively large set of training images is provided for the base class set Cb. Each training image corresponds to a tuple (x, y) consisting of an image x and an annotation y = {y0,…, yM}. Each object annotation yi = {ci, bi} contains a category label (ci) and a bounding box (xi, yi, wi, hi). After the FSOD model is trained, it will be evaluated based on a limited number (k) of training images from the Cn set that can be used for different new (i.e., few sample) categories.

[0085] Meta-adjustment process: This embodiment uses the reinforcement learning (RL) method of REINFORCE to search for the best loss function and enhancement. In order to meta-adjust the loss function and enhancement to maximize the generalization ability of FSOD, this embodiment generates proxy tasks on the base class training data to imitate the real FSOD tasks on the new class. To this end, this embodiment divides the base class into two subsets: proxy-base C p-base and proxy-novel C p-novel Then, this embodiment uses the base class training set to construct three non-overlapping data sets: (i) D p-pretrain Contains only C p-base Class samples are used to train a temporary object detection model for meta-adjustment; (ii) D p-support Contains C p-base ∪C p-novel Class samples are used as fine-tuning images during meta-adjustment; (iii) D p-query Contains C p-base ∪C p-novel Class samples are used to evaluate the generalized FSOD performance of the fine-tuned model during meta-tuning.

[0086] This embodiment generates a series of FSOD proxy tasks for meta-adjustment, similar to scenario learning: In each proxy task T, this embodiment starts from D p-support Some sample training sets are sampled from the training set. This embodiment also samples the loss function / enhancement amplitude parameter ρ, where each ρ j ∈ρ are modeled according to Gaussian distribution: Using a loss function or augmentation corresponding to the sampled ρ, this embodiment uses gradient-based optimization to fine-tune the initial model on the support image and calculate D p-query The mean average precision (mAP) score of the proposed approach is obtained by repeating this process multiple times on multiple proxy support samples. Then, the μ value is updated by the REINFORCE rule after each scenario training to find the μ value centered on the well-performing ρ combination.

[0087]

[0088] Among them, p(ρ j ;μ j ,σ) Gaussian density function, μ is the RL learning rate.

[0089] Meta-adjustment loss function: The main goal of this embodiment is to improve the fine-tuning of small-shot detectors based on the principle of meta-learning. In order to meta-adjust the FSOD loss, this embodiment pays special attention to the classification loss term, because FSOD errors are often mainly caused by misclassification.

[0090]

[0091] N ROI is the number of ROIs (i.e. candidate regions) in the image, y i is the true category label of the i-th ROI, f(x i ,y) is the corresponding prediction score of class y. ρ (t) = exp(ρ a t 2 +ρ b +ρ c ),ρ=(ρ a ,ρ b ,ρ c ) is a triplet of polynomial coefficients and t∈[0,1] is the normalized fine-tuning iteration index. The temperature is increased or decreased over time to make the predicted class distribution smoother or sharper. For y∈C b ,α(y) is set to 1, otherwise the novelty fraction scaling factor ρ α , as a way to learn the balance of basic and novel scores.

[0092] A model capable of identifying specific crops can be trained in the above manner. The model can be fine-tuned based on a small number of captured crop images and quickly fitted. It does not require thousands of specified crop images, greatly saving manpower, material and financial resources.

[0093] Step 4: The dedicated APP of the terminal subsystem includes the automatic switching solution of video streams. Figure 4 As shown in the figure, the user mode is divided into exclusive mode and shared mode. In exclusive mode, the user can control the car path and camera perspective. In shared mode, the user can only virtually control the camera perspective, which is achieved by dynamically cutting a part of the panoramic video to simulate the effect of perspective change. The camera perspective includes the car shooting perspective and the farm overlooking camera perspective. When the car is out of power or fails, it automatically switches to the farm overlooking perspective to avoid a black screen.

[0094] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A smart farm inspection live broadcast system, characterized in that: Includes field subsystem, service subsystem and terminal subsystem; The field subsystem includes: Mobile inspection module, used to conduct mobile inspections within the farm and capture multi-angle inspection video data during the inspection process; The service subsystem includes: The front-end edge computing module is used to perform panoramic stitching on the inspection video data to obtain a panoramic video; and encode the panoramic video to obtain encoded data; A business center module is used to construct a crop identification model, and execute the crop identification task of the coded data based on the crop identification model to obtain a crop identification result; The terminal subsystem includes: User terminal, used for live video streaming, video streaming playback, farm status push and user interaction command sending; The user mode control terminal is used to switch the user's control method for the mobile inspection module.

2. According to claim 1, a smart farm inspection live broadcast system is characterized in that: The mobile inspection module comprises: A navigation submodule, used to obtain environmental information and plan the mobile inspection path of the vehicle based on the environmental information; The communication submodule is used to communicate with the service subsystem and transmit control instructions and video data; The video acquisition submodule is used to capture multi-view inspection video data around the vehicle.

3. According to claim 1, a smart farm inspection live broadcast system is characterized in that: The field subsystem further includes: The fixed bird's-eye view camera module is used to capture bird's-eye view video data of the farm based on a preset position.

4. According to claim 1, a smart farm inspection live broadcast system is characterized in that: Front-end edge computing module, including: The panoramic stitching submodule is used to perform image registration, image fusion and image stitching processing on the inspection video data of each channel to obtain a panoramic video; The encoding submodule is used to perform block processing on the panoramic video to obtain a plurality of rectangular blocks. In the same frame, the pixel values ​​of the adjacent rectangular blocks of the panoramic video frame are used to predict the pixel values ​​of the current rectangular block, and the pixel values ​​corresponding to each rectangular block are quantized, encoded and filtered to obtain encoded data.

5. According to claim 1, a smart farm inspection live broadcast system is characterized in that: The business center module includes: The crop recognition model construction submodule is used to construct a crop recognition model based on a small sample target detection model; A model training submodule is used to obtain training data, wherein the training data includes training image data and corresponding crop labels, divide the training data into a proxy base class set and a proxy new class set, and construct an initial crop recognition model based on the proxy base class set; construct a non-overlapping first data set, a second data set, and a third data set based on the proxy base class set and the proxy new class set; the first data set includes samples of the proxy base class set, which are used for model training for meta-adjustment, the second data set and the third data set both include samples of the proxy base class set and the proxy new class, the second data set is used to fine-tune the image during model meta-adjustment, and the third data set is used to evaluate the performance of the fine-tuning model; Training and evaluation are performed on multiple proxy tasks based on scenario learning, specifically including: sampling a support set in a first data set, simulating a new category of proxy tasks based on the support set, sampling a loss function and an enhancement amplitude parameter, and fine-tuning an initial crop recognition model on the support set based on a gradient optimization method. During the fine-tuning process, the parameters of the initial crop recognition model are updated according to the direction of gradient descent to minimize the loss function; after the fine-tuning is completed, an average accuracy score of the initial crop recognition model is calculated based on a third data set; Repeat the scenario learning process to obtain several average accuracy scores, and use the reinforcement learning algorithm to optimize the model parameter configuration after each scenario learning; The model is trained through the scenario learning process and the model parameter configuration optimization process to obtain a trained crop recognition model; The intelligent identification submodule is used to input the encoded data into the trained crop identification model for identification prediction to obtain the corresponding crop identification results.

6. According to claim 1, a smart farm inspection live broadcast system is characterized in that: The service subsystem further includes: The artificial intelligence acceleration module is used to access the business center module and provide computing acceleration services for the crop identification algorithm.

7. The smart farm inspection live broadcast system according to claim 1, characterized in that: The user mode control terminal comprises: Exclusive module for manually controlling the mobile inspection route and camera viewing angle; Shared module for virtually changing the camera perspective.

8. The smart farm inspection live broadcast system according to claim 7 is characterized in that: The shared module includes: The dynamic cropping submodule is used to dynamically crop the panoramic video according to the user interaction data to simulate the change of viewing angle; The video stream automatic switching submodule is used to monitor the power and maintenance requirements of the mobile inspection module in real time. When the mobile inspection module needs to be offline, it automatically switches to the video source of the fixed overhead camera module; The dynamic resource allocation submodule is used to dynamically allocate mobile inspection resources according to the number of user terminals.

Images