A double-terminal crop phenotype measurement method based on dynamic management and adaptive app

Abstract

The application relates to a new method and a corresponding system for double-terminal crop phenotype measurement based on dynamic management and adaptive APP, and belongs to the technical field of augmented reality and artificial intelligence. The system comprises an AR interactive collection module, a cloud intelligent reasoning module and a cross-terminal collaborative management module. A unified configuration pulling protocol and a bidirectional synchronization mechanism are established among a management end server, a wearable augmented reality glasses terminal, an adaptive mobile terminal application and N crop phenotype calculation servers, so that on the premise that a deployed program on a client side is not recompiled, crop phenotype measurement categories can be expanded at a running time, an application interface can be self-adapted at a user level, and double-terminal data can be aligned at an account level.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality and artificial intelligence, and in particular to a novel method and system for measuring crop phenotypic patterns on dual terminals based on dynamic management and adaptive apps, which can be applied to scenarios such as crop breeding, germplasm resource identification, and field phenotypic collection. Background Technology

[0002] Accurate assessment of rice panicle traits is a core aspect of crop breeding and precise identification of germplasm resources. Key indicators such as the total number of grains in the main panicle, the length from the neck to the tip of the main panicle, and the density of spikelets on secondary branches directly determine the yield potential and varietal quality of rice. However, the rice panicle structure is complex, especially the dense arrangement and varied morphology of spikelets on secondary branches. Precise quantification of these traits not only helps in understanding yield composition but also serves as a crucial scientific basis for screening superior germplasm resources. Currently, the measurement and grading of rice panicle traits mainly rely on traditional manual measurement and visual grading methods. Technicians must use rulers or calipers to record data one by one in the field or laboratory. This method is not only time-consuming and labor-intensive but also easily affected by subjective human judgment. Especially when dealing with complex traits such as spikelet density on secondary branches, manual measurement struggles to provide accurate quantitative results, leading to a lack of standardization of data across different environments and operators, directly impacting the scientific validity of subsequent breeding decisions.

[0003] In recent years, computer vision-based digital detection technologies have provided new solutions to the aforementioned problems. For example, a rapid detection method for effective panicles in rice (CN111462058A) is disclosed in the prior art. This method uses a deep learning model to automatically count effective panicles in rice field images collected by drones. However, this method uses a traditional convolutional neural network as the backbone network, which is limited by the local convolutional receptive field and is difficult to establish long-distance semantic associations under conditions of dense grain occlusion, resulting in a high false negative rate. In addition, patent application CN113012150A also discloses a high-density rice field drone image panicle counting method with feature fusion. This method solves the problem of dense rice panicle counting through feature pyramid fusion and density estimation, but it is only for panicle number estimation in drone aerial photography scenarios and does not support fine measurement of panicle traits at close range. Moreover, the model has high computational complexity and is difficult to meet the needs of real-time inference.

[0004] It is worth noting that existing digital detection methods typically require inspectors to first take images with a mobile device, wait for the images to be uploaded to a server for analysis, and then switch to another screen to view the results. This fragmented workflow easily interrupts the inspectors' attention, greatly increases the workload, and makes it difficult to verify or retake images on-site in real time. Taking the current rice panicle seed measurement system on the market as an example, users must place the rice panicle on a specific black background, and may even need to manually disassemble and spread out the primary and secondary branches, placing the base of the panicle neck above the marked line on the background, before taking a picture with a mobile device, and then wait for the upload and analysis and switch screens to view the results. This cumbersome, discontinuous operation mode and delayed feedback mechanism not only result in high labor costs but also disrupt the smoothness of the interactive experience.

[0005] From the perspective of detection models, existing mainstream object detection methods still have significant limitations when processing rice panicle images. Traditional convolutional neural network methods are constrained by the inherent limitations of local convolutional receptive fields, making it difficult to establish long-distance semantic relationships under severe occlusion conditions. This leads to missed detections of grains and panicle branches that are far apart but structurally related. While Transformer-based methods have stronger global feature representation capabilities, the quadratic computational complexity of their self-attention mechanism significantly increases inference time on high-resolution panicle images, making them difficult to deploy in real-time interactive scenarios with strict low-latency requirements. Therefore, there is an urgent need for an object detection architecture that can establish long-range dependencies with linear computational complexity to effectively solve the problem of missed detections under dense occlusion conditions while maintaining real-time inference speed.

[0006] Current digital inspection systems generally lack the ability to be flexibly deployed for multiple users and institutions. Traditional mobile inspection applications typically use hard-coded, fixed inspection models and server addresses, resulting in limited and rigid functionality. They struggle to adapt dynamically to changing user needs, and the lack of isolation mechanisms for image data between different users poses security risks. While augmented reality glasses have seen initial applications in agricultural pest and disease surveys, hazard level diagnosis, and plant organ annotation, their use in the precise measurement of rice panicle traits remains a gap. Therefore, there is an urgent need for an intelligent inspection system that can overlay and display inspection results in real time within an augmented reality view, achieving a "what you see is what you get" experience, to overcome the multiple bottlenecks of traditional methods in terms of interactive experience, inspection accuracy, and system scalability.

[0007] At the level of large-scale deployment across users and institutions, existing crop phenotyping systems generally suffer from the following technical bottlenecks. First, existing crop phenotyping client applications typically use hard-coded binding to a single computing server network address. Once deployed, switching to another computing server requires modifying the client source code and recompiling and republishing, resulting in long deployment cycles and high maintenance costs across institutions. Existing rapid detection methods for effective panicles in rice employ a client-side architecture that directly calls a fixed backend, lacking dynamic configuration capabilities. Second, existing crop phenotyping systems typically deeply couple the image acquisition terminal with a specific crop phenotyping algorithm. When users need to expand to new crop types or new measurement indicators, the entire system must be redesigned, lacking plug-and-play expansion capabilities for new crop phenotyping measurement categories. Furthermore, existing augmented reality (AR) glasses-based crop detection solutions are only single-terminal architectures, lacking account-level bidirectional data synchronization with mobile applications. After data collection using AR glasses in the field, if personnel need to access historical records or perform batch data export at the office, they still must use separate desktop software, separating the data collection site from the post-processing stage. Finally, the application icons, background images, and function description text of the existing system are usually fixed and written during client compilation. When faced with the brand customization needs of different organizations, the developer must compile and release a customized version for each organization separately, which makes it difficult to achieve adaptive rendering of a single client program for multiple organizations and multiple users.

[0008] In summary, existing technologies have systemic shortcomings in three dimensions: dynamic scalability, cross-terminal collaboration, and interface adaptability of crop phenotyping systems. There is an urgent need for a new crop phenotyping method and system that can dynamically expand the number of crop phenotyping computing servers without recompiling the deployed client, and achieve account-level bidirectional synchronization and adaptive rendering between wearable augmented reality glasses terminals and mobile applications. Summary of the Invention

[0009] To address the four core bottlenecks in existing crop phenotyping systems—hard-coded binding of computing servers, deep coupling of single algorithms, lack of cross-terminal collaboration capabilities, and difficulty in adaptively customizing interfaces—this invention provides a novel dual-terminal crop phenotyping method and system based on dynamic management and an adaptive app. By establishing a unified configuration retrieval protocol and bidirectional synchronization mechanism among the management server, wearable augmented reality glasses terminal, adaptive mobile application, and N crop phenotyping computing servers, this invention achieves, from a fundamental perspective, the expansion of crop phenotyping measurement categories at runtime, the adaptiveness of the application interface at the user level, and the alignment of dual-terminal data at the account level, without recompiling the already deployed program on the client side.

[0010] The technical solution of this invention is as follows: a dual-terminal crop phenotyping method based on dynamic management and an adaptive APP, which is executed collaboratively by a management server, a wearable augmented reality glasses terminal, an adaptive mobile application, and N crop phenotyping calculation servers, where N is a positive integer greater than or equal to 2 and expandable during runtime. The management server stores N network addresses of crop phenotyping calculation servers, as well as interface element configuration information and function description text corresponding to each user account. Each crop phenotyping calculation server network address corresponds to a crop phenotyping measurement category. In response to an instruction to add the (N+1)th crop phenotyping calculation server network address, the management server immediately adds the new network address to its storage, allowing the client to add crop phenotyping measurement categories without recompiling. The wearable augmented reality glasses terminal and the adaptive mobile application send configuration retrieval requests to the management server based on the user account login credentials. The management server returns the crop phenotyping calculation server network address, interface element configuration information, and function description text corresponding to the user account based on the configuration retrieval request. Both terminals perform adaptive rendering of their local user interface based on the returned interface element configuration information. In response to the user's selection of a crop phenotypic measurement category and triggering of an image acquisition operation on any terminal, the selected terminal sends the acquired crop image to the corresponding crop phenotypic calculation server, which then returns the crop phenotypic measurement results. The wearable augmented reality glasses terminal overlays the crop phenotypic measurement results onto the augmented reality field of view, while the adaptive mobile application displays the crop phenotypic measurement results on the mobile interface. The crop phenotypic measurement results are bidirectionally synchronized between the two terminals using the same user account through the management server.

[0011] The present invention also provides a dual-terminal crop phenotyping system based on dynamic management and an adaptive APP, including a management server, a wearable augmented reality glasses terminal, an adaptive mobile application, and N crop phenotyping calculation servers, which are connected to each other through a computer network and collaboratively execute the above-described method.

[0012] The beneficial effects of this invention are as follows: First, the management server centrally and dynamically configures N network addresses for crop phenotyping calculation servers and N crop phenotyping measurement categories. This allows the client to add the N+1th, N+2th, or even any other network address for crop phenotyping calculation servers after deployment, through configuration and pull protocols. The mechanism lies in decoupling the strong coupling between the client, algorithm, and server into a loosely coupled triangular structure between the client, management server, and calculation server. The client only handles data collection and rendering, while the algorithm is deployed to arbitrarily expandable crop phenotyping calculation servers. This eliminates the bottleneck of traditional hard-coded architectures being unable to expand across crop categories from a fundamental perspective. Compared to a single-backend fixed architecture, this invention does not require recompiling the client when adding new crop phenotyping measurement categories. Second, the application identifier image, background image, and functional description text for adaptive mobile applications are dynamically distributed by the management server according to user accounts. This eliminates the need for separate compilation and customization for each institution, enabling a single client program to achieve branded adaptive rendering for multiple institutions and users. Third, the wearable augmented reality glasses terminal and the adaptive mobile application achieve bidirectional synchronization at the account level through the same management server. The crop phenotypic measurement results completed by either terminal appear instantly in the historical record list of the other terminal, eliminating the problem of data separation between the acquisition end and the post-processing end in existing technologies. Fourth, N crop phenotypic computing servers are independently deployed on network-isolated computing nodes. Each server only responds to user account requests authorized by the management server, achieving physical isolation between different user data at the architectural level and ensuring the privacy and security of crop phenotypic image data. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall system architecture of the present invention.

[0014] Figure 2 This is a schematic diagram of the functional interaction of a cross-terminal collaborative system.

[0015] Figure 3 This is a diagram of the management server configuration interface.

[0016] Figure 4 This is a schematic diagram of the mobile application detection results interface.

[0017] Figure 5 This is a schematic diagram illustrating the overlay display effect of the detection results from the AR glasses. Detailed Implementation

[0018] The technical solution of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0019] like Figure 1As shown, the crop phenotypic cross-terminal AR intelligent detection system provided by this invention includes an AR interactive acquisition module 1, a cloud-based intelligent inference module 2, and a cross-terminal collaborative management module 3. These three modules form a deeply coupled closed-loop collaborative relationship: the ear image acquired by the AR interactive acquisition module 1 is uploaded to the cloud-based intelligent inference module 2; the cloud-based intelligent inference module 2 calls the state-space augmented detection model to perform target detection inference and complete the quantitative calculation of ear traits; the detection result is returned to the AR interactive acquisition module 1 for real-time overlay display in the augmented reality field of view, and simultaneously transmitted to the mobile application of the cross-terminal collaborative management module 3 for users to view historical data and manage data. The management server in the cross-terminal collaborative management module 3 dynamically distributes the inference server network address and functional mode configuration to the AR interactive acquisition module 1 and the mobile application. Figure 2 As shown, the entire system constructs an efficient closed loop from data collection at the augmented reality glasses end, cloud computing to feedback at the augmented reality glasses end, and achieves seamless connection between field augmented reality screening and indoor tablet verification through cross-terminal collaborative management.

[0020] like Figure 1 As shown, the dual-terminal crop phenotyping system based on dynamic management and an adaptive APP provided by this invention includes a management server 4, a wearable augmented reality glasses terminal 5, an adaptive mobile application 6, and N crop phenotyping calculation servers 7. The management server 4 serves as the configuration and scheduling hub of the entire system. The wearable augmented reality glasses terminal 5 and the adaptive mobile application 6 are the dual-terminal data acquisition and display carriers, respectively. The N crop phenotyping calculation servers 7 correspond to N types of crop phenotyping measurement categories. In this embodiment, the initial value of N is 3. The three crop phenotyping server network addresses IP1, IP2, and IP3 correspond to the whole ear shape measurement category, the rapeseed grain counting category, and the multi-crop grain variety category, respectively. Figure 3 As shown in the management server configuration interface, administrators can add the 4th and 5th crop phenotypic calculation server network addresses at any time to expand new crop phenotypic measurement categories (such as wheat ear analysis, corn kernel testing, soybean grain type measurement, etc.). The newly added network addresses take effect immediately without the need to recompile the deployed wearable augmented reality glasses terminal program and adaptive mobile application.

[0021] like Figure 2As shown, the core functions of the management server 4 include three aspects: First, dynamically configuring N network addresses for crop phenotyping calculation servers for both terminals; second, dynamically configuring interface element configuration information such as application identification images, background images, and function introduction text for each user or organization; and third, serving as the synchronization hub for crop phenotyping measurement results between the two terminals. Upon startup, the wearable augmented reality glasses terminal 5 and the adaptive mobile application 6 send a configuration retrieval request to the management server 4 based on the user's account login credentials. The management server 4 then returns a network address configuration table, interface element configuration information, and function introduction text specific to that user's account.

[0022] The core innovation of this invention lies not in a specific crop phenotypic measurement algorithm itself, but in the software architecture that decouples the terminal, management terminal, and computing server. Any existing or future crop phenotypic measurement algorithm (including but not limited to object detection based on convolutional neural networks, object detection based on Transformer architecture, and object detection based on state space models) can be connected to this system as a new crop phenotypic computing server. Only the corresponding crop phenotypic computing server network address needs to be registered in the management terminal server 4; no code modification is required for the wearable augmented reality glasses terminal 5 and the adaptive mobile application 6. The following examples illustrate implementation methods for three typical crop phenotypic measurement categories: whole ear shape measurement, rapeseed grain counting, and multi-crop grain variety assessment, to further clarify the technical solution of this invention.

[0023] The AR interactive acquisition module 1 is deployed in a wearable augmented reality glasses device. In one embodiment of the present invention, the augmented reality glasses device adopts a lightweight head-mounted design, with a total weight of no more than 80g, and is equipped with a camera unit with a resolution of no less than 1920×1080 pixels and an augmented reality display screen with waveguide transmissive display capability. The core functions of this module include image acquisition and encoding, voice interaction control, detection function mode switching, and augmented reality field-of-view overlay rendering.

[0024] In terms of image acquisition and encoding, when a user triggers a shooting action via voice command (e.g., issuing a voice command to take a picture) or by lightly touching the side touchpad of the augmented reality glasses, the camera unit instantly captures an image of the crop ear in the current field of view. Preferably, to ensure a balance between upload efficiency and image quality, the AR interactive acquisition module 1 performs JPEG compression encoding on the captured raw image, with a compression quality coefficient ranging from 85 to 95, preferably 90 in this embodiment. The encoded image data packet is appended with metadata information such as a timestamp, user identifier, and current detection function mode identifier, and uploaded to the inference server where the cloud intelligent inference module 2 resides via HTTP protocol through a wireless network. In one embodiment of the present invention, the typical size of the image data packet is 200kB to 500kB, and the upload latency in a 4G or WiFi network environment does not exceed 300ms.

[0025] In terms of voice interaction control, the augmented reality glasses integrate a microphone array and a voice recognition engine, allowing users to perform operations such as taking photos, switching functions, and confirming results using natural language commands without using their hands. Preferably, the voice recognition engine supports a preset set of command words, including but not limited to commands such as "take photo," "switch mode," and "next," with a recognition response time of no more than 500ms. This hands-free interaction method is particularly suitable for the demands of working in complex field conditions, completely freeing the hands of inspection personnel.

[0026] Regarding the switching of detection function modes, the AR interactive acquisition module 1 includes a mode switching unit, which responds to the user's swipe gestures or voice commands to switch between multiple detection function modes. In one embodiment of the present invention, the multiple detection function modes include a whole panicle shape measurement mode, a grain counting mode, and a multi-crop testing mode. The whole panicle shape measurement mode is used for fine measurement of panicle traits in rice, including the total number of grains in the main panicle, panicle length, and branch statistics; the grain counting mode is suitable for rapid grain counting in multiple crops such as wheat, corn, and soybeans; the multi-crop testing mode supports morphological parameter analysis of grains from multiple crops. Each detection function mode corresponds to a different cloud inference server network address and detection model. When switching modes, the AR interactive acquisition module 1 automatically sends an analysis request to the corresponding inference server.

[0027] In terms of augmented reality (AR) view overlay rendering, after the cloud-based intelligent inference module 2 completes inference analysis and returns the detection result data, the rendering engine of the AR interactive acquisition module 1 overlays the detection results onto the user's AR view in real time. Preferably, the rendering engine displays the labeled detection image in the central area of ​​the AR view, while simultaneously presenting structured trait measurement data in a floating table format above the labeled image, including information such as total grain count, spike length, number of primary branches, number of secondary branches, and spikelet density level. Figure 5As shown, users can directly obtain complete detection and grading results within their field of vision without looking down at other devices, truly achieving an immersive detection experience where what you capture is what you get. In one embodiment of the present invention, the end-to-end response delay from the completion of shooting to the display of the complete detection result in the augmented reality field of vision does not exceed 2 seconds, and the actual measured delay is approximately 1.8 seconds.

[0028] Furthermore, regarding the standardization of image acquisition conditions, to ensure the stability and repeatability of detection accuracy, this invention sets standardized requirements for the acquisition conditions of the panicle image. Preferably, the inspector places the rice panicle to be tested on a neutral gray background paper for photographing. The grayscale value of the neutral gray background paper is approximately 128, which serves to provide a uniform and low-contrast background to reduce the interference of complex background textures on target detection. In one embodiment of this invention, the physical size of the neutral gray background paper is A3 size (297mm × 420mm), and its edges are pre-printed with calibration lines of known spacing. The spacing between the calibration lines is 10cm, which is used to provide a reference benchmark for calculating the pixel-to-physical scale conversion coefficient. During shooting, the recommended distance between the camera unit of the augmented reality glasses and the panicle is 20cm to 40cm. Within this distance range, it can be ensured that the panicle target occupies a sufficient pixel area in the image to achieve fine detection. In addition, the AR interactive acquisition module 1 also integrates an image quality assessment function. After shooting, it automatically detects the blur level and exposure uniformity of the image. When the image quality is lower than the preset threshold, it prompts the user to retake the shot through voice, thereby ensuring the quality of the input data at the source.

[0029] The state-space augmented detection model is the core model for achieving high-precision panicle target detection in this invention. This model runs on the inference server hosted by the cloud-based intelligent inference module 2. It embeds the state-space model into the backbone network of the YOLO target detection architecture, establishing long-range dependencies in the input feature sequence through the linear complexity recursive mechanism of the state-space model. This effectively addresses challenges commonly encountered in rice panicle images, such as dense grain occlusion, drastic scale changes, and blurred panicle branch structure.

[0030] In terms of overall architecture, the state-space augmented detection model adopts the YOLO series detection framework as its basic architecture, which consists of three components: a backbone network, a neck feature fusion network, and a detection head. The core innovation of this invention lies in replacing some convolutional modules in the traditional YOLO backbone network with state-space model modules, forming a hybrid backbone network with state-space augmentation. This hybrid backbone network retains the efficient extraction capability of convolutional operations for local texture features while introducing the state-space model's ability to perceive global long-range context; the two complement and synergize with each other.

[0031] Regarding the core recursive mechanism of the state-space model, it discretizes the state transition equations of a continuous-time system into a recursive computation form that can be efficiently executed on digital hardware. Specifically, the state transition equations of the continuous-time state-space model are:

[0032]

[0033] in: for The hidden state vector at time step , with dimension . , The value range is from 16 to 128, and in this embodiment, 64 is preferred. The selection criteria are based on achieving a balance between long-range memory retention and computational overhead, with larger... Values ​​can encode richer contextual information but increase the number of parameters; for The input features at time step, with dimension 1 , Corresponding to the number of feature channels, in this embodiment The value is 256; for The output features at time step 1, also have the same dimension. ; The state transition matrix has dimension 1. To control the temporal evolution of the hidden state; The input projection matrix has dimensions of . This maps the input features to the hidden state space. To output the projection matrix, the dimension is... This maps the hidden state back to the output feature space.

[0034] To enable the aforementioned continuous system to be executed on a discrete computing device, this invention employs the zero-order hold method to discretize the state transition equations. A step size parameter is introduced during the discretization process, with a value ranging from 0.001 to 0.1. In this embodiment, a value of 0.01 is preferred. The advantage of this parameter is that it controls the temporal resolution of the continuous-to-discrete transition; a smaller value preserves finer-grained temporal information but increases the number of computational steps. The discretized recursive equation is:

[0035] in: For the first The discrete hidden state vector of the step; This is the hidden state vector from the previous step; For the first The input feature vector of the step; For the first The output feature vector of the step; The discretized state transition matrix is ​​calculated as follows: ; The discretized input projection matrix is ​​calculated as follows: ,in for The identity matrix.

[0036] Preferably, the state transition matrix Initialization is performed using the HiPPO initialization method. HiPPO initialization involves... The elements are set to a specific polynomial basis coefficient matrix form, which allows the hidden state to compress and preserve historical input information in a near-optimal manner during the recursive process, thereby significantly enhancing the model's ability to model long-range dependencies. This characteristic is crucial for modeling the semantic associations between grains and panicle structures that are far apart in rice panicle images.

[0037] Regarding the selective state scanning mechanism, directly applying one-dimensional state space recursion to a two-dimensional image feature map requires a reasonable unfolding strategy. This invention employs a selective state scanning mechanism, unfolding the two-dimensional feature map into a one-dimensional sequence along four directions, and performing state space recursion on each direction. These four directions include left-to-right, right-to-left, top-to-bottom, and bottom-to-top. Preferably, the scanning path in each direction independently performs state space discretization recursion, generating four sets of directional feature outputs. Subsequently, the recursive outputs from the four directions are fused element-wise to obtain a feature representation with a global receptive field. This multi-directional scanning strategy ensures that an effective information propagation path can be established between any two positions in the feature map, thereby achieving global context awareness without introducing quadratic computational complexity. In one embodiment of this invention, on an NVIDIA A10 graphics processing unit, the model's inference speed can reach 28 frames per second, meeting real-time inference requirements.

[0038] Regarding occlusion perception confidence correction, to address the false detection problem caused by dense occlusion of grains in rice panicle images, the state-space augmentation detection model further performs occlusion perception confidence correction when outputting detection results. Specifically, for any detection box... First, calculate the cross-union ratio (CUB) between the detection box and its surrounding adjacent detection boxes, and use the maximum CUB value as the occlusion ratio of that detection box. , The value ranges from 0 to 1, where 0 represents no occlusion and 1 represents complete occlusion. The corrected detection confidence score. The calculation formula is:

[0039] in: The detection confidence score of the original output of the state space enhancement detection model is set to a value ranging from 0 to 1. The occlusion penalty coefficient ranges from 0.1 to 0.5, and is preferably 0.3 in this embodiment. The technical effect lies in controlling the penalty of occlusion degree on confidence level, with a larger... This value makes the model more sensitive to occlusion, which helps suppress false detections in highly occluded areas but may increase the risk of missed detections. A value of 0.3 achieves the optimal balance between false positive and false negative rates. This represents the occlusion ratio of the current detection bounding box. The corrected detection confidence score... Below the preset threshold At that time, suppress the output of the corresponding detection box. The value ranges from 0.3 to 0.7, and 0.5 is preferred in this embodiment. This occlusion perception correction mechanism improves the detection accuracy of the model in densely occluded scenes by about 3 to 5 percentage points compared to the uncorrected version.

[0040] In terms of detection head design, the state-space augmented detection model adopts a decoupled detection head structure, separating the classification prediction branch and the localization regression branch into two independent convolutional channels. The classification prediction branch outputs the probability distribution of each candidate position belonging to each target category. In this embodiment, the target categories include four categories: grains, primary branches, secondary branches, and ear necks. The localization regression branch outputs the detection box boundary offset corresponding to each candidate position, and uses distributed focal loss for supervised training to improve boundary localization accuracy. Preferably, the detection head performs predictions at three different feature levels with corresponding downsampling rates of 8x, 16x, and 32x, respectively, to cover multi-scale targets from small grains to whole ear structures. The 8x downsampling feature level is mainly responsible for detecting grain targets with small pixel areas, the 16x downsampling feature level is responsible for detecting medium-scale branch targets, and the 32x downsampling feature level is responsible for detecting larger-scale ear neck targets.

[0041] Regarding the model training strategy, the first stage involves pre-training the backbone network on a large-scale general object detection dataset to obtain general visual feature representation capabilities. The second stage involves end-to-end fine-tuning on a rice panicle trait-specific dataset constructed in this invention. During fine-tuning, a cosine annealing scheduling strategy is used, with an initial learning rate of 0.01, a total of 300 training epochs, and a batch size of 16. During training, strategies such as random horizontal flipping, random scaling (ranging from 0.5 to 1.5), and Mosaic data augmentation are employed to improve the model's generalization ability.

[0042] The cloud-based intelligent inference module 2 is deployed on a cloud-based inference server, undertaking the core inference tasks of image reception, model invocation, quantification calculation of ear traits, and result encapsulation and feedback. For example... Figure 1 As shown, a two-way data channel is established between the cloud-based intelligent inference module 2, the AR interactive acquisition module 1, and the cross-terminal collaborative management module 3.

[0043] In terms of image reception and preprocessing, the cloud-based intelligent inference module 2 deploys a RESTful application programming interface based on the Flask framework to receive JPEG format image data packets uploaded by the AR interactive acquisition module 1 or a mobile application. Upon receiving the image data packets, the cloud-based intelligent inference module 2 first parses the metadata information to determine the current detection function mode, and then performs size normalization processing on the image, scaling the input image to a fixed input size required by the detection model. In one embodiment of the invention, the fixed input size is 640×640 pixels. The preprocessed image tensor is then fed into the state-space augmented detection model for forward inference.

[0044] In terms of quantifying ear traits, the cloud-based intelligent inference module 2 performs quantification calculations of multidimensional trait parameters based on the detection results output by the state-space enhanced detection model. The detection results include the detection box coordinates, category identifier, and detection confidence score for each detected target. In one embodiment of the present invention, the detection categories include four types of targets: grains, primary branches, secondary branches, and ear neck nodes.

[0045] For calculating the total number of grains in the main ear, the cloud-based intelligent inference module 2 counts the number of detection frames that are all classified as grains and whose corrected detection confidence score is higher than a preset threshold. This number represents the total number of grains in the main ear.

[0046] For calculating the length from the neck node to the tip of the main panicle, the cloud-based intelligent inference module 2 first locates the center coordinates of the detection box categorized as the neck node as the reference point for the base of the panicle. Then, it searches for the position corresponding to the minimum vertical coordinate value in all grain detection boxes as the reference point for the tip of the panicle, and calculates the pixel distance between the two reference points. Subsequently, the pixel-to-physical scale conversion coefficients were used. Convert pixel distance to physical distance, ear length measurement. The calculation formula is:

[0047] in: The length from the neck node to the tip of the main panicle, in cm; This is the pixel distance from the reference point at the neck of the ear to the reference point at the top of the ear, measured in pixels, and is calculated using the Euclidean distance between the two points. This is a pixel-to-physical scale conversion factor, measured in cm / pixel, determined based on the ratio of the known physical size of a reference calibration object (in this embodiment, a neutral gray background paper edge calibration line with known physical size) in the image to its corresponding pixel size. In one embodiment of the invention, The typical value range is 0.01 to 0.05 cm / pixel, with the specific value depending on the shooting distance and lens focal length. Preferably, the physical distance measurement accuracy of the ear length is not less than 0.5 cm, and the actual measurement accuracy can reach 0.3 cm.

[0048] For calculating the spikelet density on secondary branches, the cloud-based intelligent inference module 2 counts the spikelet targets on the secondary branches based on the detection boxes output by the state-space augmented detection model, thus obtaining the spikelet count. Simultaneously, the length of the secondary branch is calculated based on the coordinates of the endpoint key points of the secondary branch detection box. The unit is cm. Density of spikelets on secondary branches. The calculation formula is:

[0049] middle: The density of spikelets on secondary branches is expressed as spikelets / cm. The number of spikelets detected on a single secondary branch is expressed in units of individual spikelets. The physical length of this secondary branch, in cm, is calculated by multiplying the pixel distance of the keypoint by... Obtained through conversion.

[0050] The cloud-based intelligent inference module 2 grades the spikelet density values ​​of secondary branches according to a preset density threshold range. In this invention, the density grading standard is set as follows: when... When the number is less than 3 / cm, it is judged as sparse. A value between 3 and 6 per cm is considered medium grade. A density level is defined as having more than 6 particles per cm.

[0051] In terms of result encapsulation and feedback, the cloud-based intelligent inference module 2 encapsulates all trait parameters into structured data in JSON format, and simultaneously generates a visual image with detection box annotations. The structured data includes fields covering the status indicator of successful detection, the total number of grains in the main ear, the ear length measurement and its unit, the number of primary branches, the number of secondary branches, the secondary branch spikelet density value and its grading results, the access path to the annotated image, and the inference time. Different categories of detection targets are identified by detection boxes of different colors in the annotated visual image. In one embodiment of this invention, the grain detection box is marked in red, the primary branch detection box in green, the secondary branch detection box in blue, and the ear neck detection box in yellow, allowing users to intuitively identify various ear structures in an augmented reality view. The structured data and annotated images are fed back to the AR interactive acquisition module 1 via HTTP response, with end-to-end inference latency controlled within 1.5 seconds. Simultaneously, the result data is synchronously transmitted to the mobile application of the cross-terminal collaborative management module 4, ensuring data consistency between the two terminals. Preferably, after completing the inference, the cloud-based intelligent inference module 3 will also record the server resource usage of this inference, including the graphics processing unit's video memory usage and inference time. This operation and maintenance data will be regularly summarized to the management server so that the administrator can monitor the system's operational health status.

[0052] In this invention, the system adopts an independent deployment architecture of one model, one server, meaning that each detection function corresponds to a separate cloud-based inference server instance. The inference server corresponding to the whole panicle shape measurement mode deploys a rice panicle trait detection model, the inference server corresponding to the grain counting mode deploys a general grain detection model, and the inference server corresponding to the multi-crop testing mode deploys a multi-crop testing analysis model. This isolated deployment architecture ensures that data from different users does not interfere with each other, thus protecting the privacy and security of image information.

[0053] like Figure 2 and Figure 3 As shown, the cross-terminal collaborative management module 3 includes two subsystems: a mobile application and a management server. It is responsible for data synchronization, function configuration, and user management between the augmented reality glasses and the mobile tablet.

[0054] On the management server side, built on the Flask framework, it provides centralized user management and dynamic configuration capabilities. Administrators use the management server's web management interface to enter network configuration information for each user or organization, such as the IP address, port number, and interface path of the cloud inference server for the deployed models. For example... Figure 3 As shown, the management interface presents all configured detection function entries in a list format. Each entry includes fields such as function name, application identifier image, inference server IP address, and function description text. Administrators can add, modify, or delete configuration entries at any time.

[0055] Preferably, the management server also dynamically configures the interface elements and functional description text of the mobile application for each user. Interface elements include the application logo image and background image, while the functional description text explains the applicable scenarios and operation methods for each detection function to the user. This dynamic configuration mechanism allows administrators to quickly customize exclusive application interfaces and function combinations for different users without recompiling the mobile application code, significantly shortening the deployment cycle from model training to actual implementation.

[0056] Regarding global configuration on the augmented reality (AR) glasses side, the management server provides a global configuration interface for AR glasses. Administrators use this interface to set the base address of the algorithm server to which all AR glasses devices connect. Upon startup, the AR glasses application automatically retrieves this base address and a list of available detection function modes from the management server, eliminating the need for users to manually enter the server address.

[0057] On the mobile application side, the mobile application is deployed on Android mobile tablet devices and shares the same configuration data with the augmented reality glasses on the same management server. After a user logs into the mobile application, the system automatically retrieves the inference server address, interface elements, and function list configured by the administrator for that user from the management server, and automatically loads the configured interface elements and function description text to achieve a customized application interface for the user.

[0058] like Figure 4 As shown, the core functions of the mobile application include using detection algorithms, viewing and exporting historical records, and batch management. Users can access the algorithm interface configured by the administrator, capture images using the mobile device's camera function, or select existing images from their local album and upload them to the cloud-based intelligent inference module 2. The uploaded image data will be automatically routed to the user's dedicated inference server for analysis. After the server completes the analysis, the results are sent back to the mobile application for the user to view instantly. Simultaneously, all detection results support categorized viewing and batch export by detection function type, with export formats including CSV data files and annotated image files.

[0059] Regarding cross-terminal data synchronization, the augmented reality glasses and mobile application of the AR interactive acquisition module 1 achieve bidirectional synchronization of detection results based on the same user account system. Each detection result executed on the augmented reality glasses is automatically synchronized to the history list of the mobile application, and the detection results executed by the user on the mobile application can also be viewed in the history list of the augmented reality glasses. In this invention, the history list supports categorization and filtering by detection function mode, such as... Figure 4As shown, users can view historical testing records for different categories, such as rapeseed grain count, grain count for multiple varieties, and rice ear detection. All historical data is synchronized in real time between the augmented reality glasses and mobile tablet according to the user's account, ensuring data integrity and traceability.

[0060] Regarding data security and user isolation, the cross-terminal collaborative management module 3 ensures data privacy through multi-layered security mechanisms. First, an authentication mechanism is used when users log in. After verifying user credentials, the management server issues an access token containing the user's identifier and validity period. The augmented reality glasses and mobile applications carry this access token in every subsequent data request to prove identity legitimacy. Second, each user's image data and detection results are stored in logically isolated data spaces, ensuring data is not visible to different users. Third, the cloud-based inference server adopts a one-model-one-server independent deployment architecture. Each inference server instance only processes requests from its assigned user, achieving physical isolation of inference computation at the network layer.

[0061] Furthermore, the cross-terminal collaborative management module 3 also provides data export and batch management functions. Users can select a specified time range or a specified detection function category for batch export of historical records in the mobile application. The export format supports CSV data files and compressed image file packages. The CSV file contains the timestamp of each detection record, the detection function mode, the values ​​of all trait parameters, and the storage path of the labeled images. The batch management function also supports batch deletion and batch labeling of historical records, making it convenient for users to efficiently organize large amounts of detection data.

[0062] To verify the technical effectiveness of the crop phenotypic cross-terminal AR intelligent detection system of the present invention, this embodiment conducted system integration and performance verification in the following test environment.

[0063] In terms of hardware, the augmented reality glasses are lightweight AR glasses devices equipped with a Qualcomm Snapdragon XR1 processor and a 1920×1080 resolution camera; the cloud inference server is equipped with an NVIDIA A10 graphics processing unit with 24GB of video memory, paired with an Intel Xeon processor and 64GB of RAM; the mobile tablet is a 10.1-inch Android tablet device.

[0064] Regarding the dataset, the test dataset contains 1200 images of rice panicles, covering panicle samples from different varieties, maturity levels, and shooting angles. Each image is manually labeled with the location information of grains, primary branches, secondary branches, and panicle neck nodes, and records the true values ​​of traits such as panicle length, number of grains, and number of branches measured manually by experts.

[0065] In terms of detection accuracy, the state-space augmented detection model achieved a mean accuracy (mAP@0.5) of 89.3% on the aforementioned test dataset, a 6.1 percentage point improvement over the baseline YOLOv8 model. In densely occluded grain scenes (the subset of samples with an occlusion ratio greater than 0.5), the recall rate of state-space augmented detection module 2 improved by 8.7 percentage points compared to YOLOv8, validating the significant performance enhancement effect of the state-space model's long-range dependency modeling capability on densely occluded scenes. Simultaneously, state-space augmented detection module 2 reduced the number of parameters by approximately 45% and the computational cost (FLOPS) by approximately 50% compared to YOLOv8, significantly reducing computational resource consumption while maintaining accuracy advantages.

[0066] Regarding counting accuracy, the correlation coefficient between the automatic counting result and the manual counting result of the total number of grains in the main ear reached 0.96, with an average absolute error of 3.2 grains. The average absolute error between the automatic measurement value and the manual vernier caliper measurement value of the ear length was 0.28 cm, with a measurement accuracy of 0.3 cm, meeting the requirement of an accuracy of not less than 0.5 cm in breeding practice.

[0067] Regarding system response latency, the average end-to-end latency from when the user triggers a capture at the augmented reality glasses to when the complete annotation and grading results are displayed in the augmented reality field of view is 1.8 seconds. This includes approximately 50 milliseconds for image compression encoding, approximately 300 milliseconds for wireless network transmission, approximately 1200 milliseconds for model inference, and approximately 250 milliseconds for result rendering. On the NVIDIA A10 graphics processing unit, the model inference speed is 28 frames per second, meeting the requirements of real-time applications.

[0068] In terms of cross-terminal collaboration, the data synchronization latency between the augmented reality glasses and the mobile application does not exceed 500ms, and the configuration distribution response time of the management server does not exceed 200ms. During stress testing with 50 concurrent users, the overall system throughput remained stable, and no inference server response timeouts occurred.

[0069] In terms of multi-crop, multi-scenario scalability verification, besides the rice panicle trait detection scenario, the system architecture of this invention is also applicable to phenotypic detection tasks for other crops. In one embodiment of this invention, by adding configuration entries and deploying corresponding inference server instances on the management server, the system successfully extended four detection functions—rapeseed grain counting, wheat panicle analysis, corn grain identification, and soybean grain shape measurement—without modifying the augmented reality glasses and mobile application code. Taking rapeseed grain counting as an example, the inspector spreads rapeseed grains on a white background paper, takes a picture through augmented reality glasses, and uploads it to the corresponding inference server. The counting results and labeled images returned by the server are superimposed and displayed in the augmented reality field of view within 1.5 seconds. Figure 4As shown, the results of different detection functions are displayed and managed separately by category in the mobile application. This practical verification result demonstrates that the dynamic configuration mechanism of the management terminal and the independent deployment architecture of one model per server of the present invention have good horizontal scalability and can quickly adapt to new detection requirements.

[0070] In comparison with existing technologies, a user experience comparison test was conducted between the system of this invention and a traditional workflow involving handheld mobile device photography followed by backend analysis. Ten technicians with over three years of field surveying experience were invited to test the same batch of 50 rice panicle samples using both the system of this invention and a traditional mobile detection system. The test results showed that using the system of this invention, the average time for technicians to complete the testing of all 50 samples was 45 minutes, while the average time using the traditional system was 82 minutes, representing an efficiency improvement of approximately 45%. During the testing process, technicians using the system of this invention reported an average of 3 attention interruptions, while those using the traditional system reported an average of 27 attention interruptions, verifying the significant improvement in workflow continuity achieved by real-time augmented reality overlay display.

[0071] The crop phenotypic cross-terminal AR intelligent detection system of this invention achieves significantly better nonlinear efficiency than independent operation of each module through the deep coupling and synergy of AR interactive acquisition module 1, cloud intelligent inference module 2, and cross-terminal collaborative management module 3. Specifically, AR interactive acquisition module 1 eliminates the disconnect between device switching and result viewing in traditional detection methods, but a standalone augmented reality acquisition terminal lacks the ability to analyze ear traits; cloud intelligent inference module 2 provides sufficient computing resources and completes multi-dimensional quantitative calculations of ear traits, but standalone cloud inference lacks end-side acquisition entry; cross-terminal collaborative management module 3 enables flexible deployment capabilities for multiple users and scenarios and data security isolation, but a standalone management system lacks detection functions. The synergistic effect of the three modules enables the system to achieve an end-to-end closed loop from field augmented reality acquisition, cloud state space augmented inference to real-time result overlay feedback, and cross-terminal data management. The output of each module serves as a key input for downstream modules, forming an inseparable organic whole.

[0072] From a technical perspective, the synergy of the three modules mentioned above brings three levels of non-linear efficiency gains: The first level is efficiency gains at the interaction level, where the hands-free acquisition capability of the augmented reality glasses and the rapid response capability of cloud inference work together to enable users to complete phenotypic detection without interrupting field operations, improving efficiency by about 45% compared to traditional methods; the second level is efficiency gains at the algorithm level, where the long-range dependency modeling capability of the state-space model and the synergy of the multi-scale detection head enable the model to improve average accuracy by more than 6 percentage points while reducing the number of parameters by 45% and the amount of computation by 50%; the third level is efficiency gains at the system level, where the dynamic configuration mechanism on the management end and the independent deployment architecture of one model per server work together to enable the system to flexibly expand new detection functions without modifying the client code, realizing a new interactive paradigm of crop phenotypic detection where what you see is what you get.

[0073] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A dual-terminal crop phenotypic measurement method based on dynamic management and an adaptive APP, characterized in that, The method is executed collaboratively by a management server, wearable augmented reality glasses, an adaptive mobile application, and N crop phenotypic calculation servers, where N is a positive integer greater than or equal to 2 and expandable during runtime. The method includes: S1, the management server stores N crop phenotypic calculation server network addresses and interface element configuration information and function description text corresponding to each user account. Each crop phenotypic calculation server network address corresponds to a crop phenotypic measurement category. The interface element configuration information includes an application identifier image and a background image. S2, in response to the instruction to add the N+1th crop phenotypic calculation server network address, the management server immediately adds the N+1th crop phenotypic calculation server network address to the storage area of ​​the management server, so that the wearable augmented reality glasses terminal and the adaptive mobile application can add crop phenotypic measurement categories without recompiling the deployed programs, and different crop phenotypic calculation servers perform different crop phenotypic category data inference; S3, the wearable augmented reality glasses terminal and the adaptive mobile application respectively send configuration pull requests to the management server based on the user account login credentials. The management server returns the network address of the crop phenotypic calculation server corresponding to the user account, the interface element configuration information, and the function introduction text according to the configuration pull request. S4, the wearable augmented reality glasses terminal and the adaptive mobile application respectively perform adaptive rendering of the local user interface according to the returned interface element configuration information, and present the crop phenotypic measurement function list according to the returned crop phenotypic calculation server network address; S5, in response to the user selecting a crop phenotypic measurement category and triggering an image acquisition operation on the wearable augmented reality glasses terminal or the adaptive mobile application, the selected terminal sends the acquired crop image to the crop phenotypic calculation server corresponding to the crop phenotypic measurement category, and the crop phenotypic calculation server returns the crop phenotypic measurement result to the selected terminal; S6, the wearable augmented reality glasses terminal overlays and displays the crop phenotypic measurement results on the augmented reality field of view, the adaptive mobile application displays the crop phenotypic measurement results on the mobile interface, and the crop phenotypic measurement results are bidirectionally synchronized between the wearable augmented reality glasses terminal and the adaptive mobile application through the management server under the same user account.

2. The method according to claim 1, characterized in that, The initial value of N is 3. The crop phenotypic measurement categories corresponding to the network addresses of the three crop phenotypic calculation servers are, in order, whole ear shape measurement category, rapeseed grain count category, and multi-crop grain test category. The crop phenotypic calculation servers corresponding to the three crop phenotypic measurement categories are deployed independently.

3. The method according to claim 1, characterized in that, When the management server receives an instruction to add a new crop phenotypic calculation server network address (N+1th), the management server binds a new crop phenotypic measurement category identifier and a new application identifier image to the new crop phenotypic calculation server network address. The new crop phenotypic measurement category identifier and the new application identifier image are returned together when the next configuration fetch request sent by the wearable augmented reality glasses terminal and the adaptive mobile application reaches the management server.

4. The method according to claim 1, characterized in that, The N crop phenotyping servers are independently deployed on computing nodes that are isolated from each other in the network. Each crop phenotyping server only responds to requests initiated by user accounts authorized by the management server.

5. The method according to claim 1, characterized in that, When performing adaptive rendering, the adaptive mobile application and the wearable augmented reality glasses terminal display the application identifier image at the entry icon position of the crop phenotyping function and the background image in the background layer of the crop phenotyping function list interface.

6. The method according to claim 1, characterized in that, After the wearable augmented reality glasses terminal completes a crop phenotypic measurement, the crop phenotypic measurement result is first uploaded to the historical record storage area associated with the management server. When the adaptive mobile application enters the historical record interface, it retrieves the result from the historical record storage area based on the same user account. When the adaptive mobile application completes a crop phenotypic measurement, it is synchronized to the wearable augmented reality glasses terminal in a symmetrical manner.

7. The method according to claim 1, characterized in that, The wearable augmented reality glasses terminal responds to the user's voice command or touch operation to trigger image acquisition. The acquired crop image is compressed and encoded and then sent to the selected crop phenotypic calculation server via HTTP request. After receiving the crop phenotypic measurement results, the wearable augmented reality glasses terminal overlays and displays the crop phenotypic measurement results on a predetermined display area of ​​the augmented reality field of view.

8. The method according to claim 2, characterized in that, The inference process executed by the crop phenotypic calculation server corresponding to the whole ear shape measurement category includes: using a target detection architecture with an embedded state space model to perform target detection on the crop ear image, and outputting the target detection box coordinates, category identifier and detection confidence; and calculating the total number of grains in the main ear, the length from the neck node of the main ear to the top of the ear, and the spikelet density of the secondary branches based on the target detection box coordinates.

9. The method according to claim 2, characterized in that, The inference process executed by the crop phenotype calculation server corresponding to the rapeseed grain count category includes: performing target detection and density estimation on the rapeseed grain image spread on the background plate, and outputting the total number of rapeseed grains.

10. The method according to claim 2, characterized in that, The inference process executed by the crop phenotypic calculation server corresponding to the multi-crop grain classification includes: performing target detection and category classification on crop grain images containing at least one of rice, wheat, corn, and soybean, and outputting the grain quantity of each crop respectively.

Images