Plant growth stage detection method and device, medium and program product

Abstract

The invention discloses a plant growth stage detection method and device, a medium and a program product, and relates to the technical field of big data and the field of financial science and technology. The method comprises the following steps: determining video features corresponding to a to-be-detected video and probability information of the to-be-detected video; according to the probability information, determining a candidate time interval in the to-be-detected video, the confidence of the candidate time interval being greater than a preset confidence threshold; the video features, the probability information and the candidate time interval are input into a target neural network, a detection result output by the target neural network is obtained, and the target neural network is used for determining the detection result in combination with plant growth prior information. The detection result comprises at least one target time interval and a plant growth stage corresponding to each target time interval. The detection result of the plant growth stage detection method provided by the embodiment of the invention is relatively high in accuracy.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular, to a method, device, medium and program product for detecting plant growth stages. Background Art

[0002] In the financial field, financial institutions can provide agricultural financial services for agricultural practitioners or agricultural institutions. For example, agricultural loans and the like. Agricultural financial services can improve agricultural production conditions, increase the yield and quality of agricultural products, and thus increase the income of agricultural practitioners or agricultural institutions. Regularly checking the growth of plants such as crops is an important part of agricultural financial services. By regularly checking the growth of crops, problems can be discovered in a timely manner and measures can be taken to ensure that agricultural financial services can be effectively used in agricultural production.

[0003] In the related art, a deep learning model can be used to detect plant growth stages. The specific process is as follows: Collect videos during the plant growth process, input the videos into a deep learning model such as a convolutional neural network, and obtain the detection results output by the deep learning model.

[0004] However, since plant growth is a continuous and uninterrupted process and the plant growth boundaries are relatively blurred, when using existing deep learning models for detection, the accuracy of the detection results is relatively low. Summary of the Invention

[0005] The present invention provides a method, device, medium and program product for detecting plant growth stages, so as to solve the technical problem of relatively low accuracy existing in the related art of plant growth stage detection methods.

[0006] According to one aspect of the present invention, a method for detecting plant growth stages is provided, and the method includes:

[0007] Determine the video features corresponding to the video to be detected and the probability information of the video to be detected; wherein, the video to be detected includes plant growth information, and the probability information is used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected;

[0008] According to the probability information, determine the candidate time interval in the video to be detected; wherein, the confidence level of the candidate time interval is greater than a preset confidence level threshold, and the confidence level of the candidate time interval is a value determined according to the corresponding boundary probability and action probability;

[0009] Input the video features, the probability information, and the candidate time intervals into a target neural network to obtain a detection result output by the target neural network; wherein, the target neural network is used to determine the detection result by combining prior information on plant growth, and the detection result includes: at least one target time interval and the corresponding plant growth stage for each target time interval.

[0010] According to another aspect of the present invention, there is provided a device for detecting plant growth stages, the device comprising:

[0011] A first determination module, configured to determine video features corresponding to a video to be detected and probability information of the video to be detected; wherein, the video to be detected includes plant growth information, and the probability information is used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected;

[0012] A second determination module, configured to determine a candidate time interval in the video to be detected according to the probability information; wherein, the confidence level of the candidate time interval is greater than a preset confidence threshold, and the confidence level of the candidate time interval is a value determined according to the corresponding boundary probability and action probability;

[0013] A third determination module, configured to input the video features, the probability information, and the candidate time interval into a target neural network to obtain a detection result output by the target neural network; wherein, the target neural network is used to determine the detection result by combining prior information on plant growth, and the detection result includes: at least one target time interval and the corresponding plant growth stage for each target time interval.

[0014] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the method for detecting plant growth stages according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided a computer-readable storage medium, the computer-readable storage medium storing a computer program, and when the computer program is used to be executed by a processor, it implements the method for detecting plant growth stages according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, there is provided a computer program product including a computer program which, when executed by a processor, implements the plant growth stage detection method according to any embodiment of the present invention.

[0020] The technical solution of the embodiment of the present invention includes: determining the video features corresponding to the video to be detected and the probability information of the video to be detected, where the video to be detected includes plant growth information, and the probability information is used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected; determining the candidate time intervals in the video to be detected according to the probability information, where the confidence level of the candidate time intervals is greater than a preset confidence threshold, and the confidence level of the candidate time intervals is a value determined according to the corresponding boundary probability and action probability; inputting the video features, the probability information, and the candidate time intervals into a target neural network to obtain the detection result output by the target neural network, where the target neural network is used to determine the detection result in combination with the prior information of plant growth, and the detection result includes: at least one target time interval and the corresponding plant growth stage of each target time interval. In this plant growth stage detection method, the target neural network can comprehensively determine the detection result based on the video features, the probability information, the candidate time intervals, and the prior information of plant growth. The probability information can characterize the action boundary information in the video to be detected, effectively alleviating the influence of the fuzzy plant growth boundary on the detection result. The prior information of plant growth can characterize the prior knowledge of plant growth, solving the problem of misjudgment caused by the detection algorithm in the related art ignoring the plant development law. Therefore, the accuracy of the detection result of the plant growth stage detection method provided by this embodiment is relatively high.

[0021] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 is a flowchart of a plant growth stage detection method provided by an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram for determining video features and probability information in an embodiment of the present invention;

[0025] Figure 3It is a flowchart of another plant growth stage detection method provided by an embodiment of the present invention;

[0026] Figure 4 It is a flowchart of yet another plant growth stage detection method provided by an embodiment of the present invention;

[0027] Figure 5 It is a schematic structural diagram of a target neural network in an embodiment of the present invention;

[0028] Figure 6 It is a schematic structural diagram of a plant growth stage detection device provided by an embodiment of the present invention;

[0029] Figure 7 It is a schematic structural diagram of an electronic device for implementing the plant growth stage detection method according to an embodiment of the present invention. Detailed implementation manners

[0030] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the term "including" and any deformation thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the embodiments of the present invention all comply with the relevant regulations of laws and regulations.

[0032] Figure 1 It is a flowchart of a plant growth stage detection method provided by an embodiment of the present invention. This embodiment is applicable to the scenario of detecting the growth stage of plants. This method can be executed by a plant growth stage detection device, and the plant growth stage detection device can be implemented in the form of hardware and / or software. The plant growth stage detection device can be configured in an electronic device, for example, a computer device. As Figure 1As shown in the figure, the method includes the following steps 101 to 103.

[0033] Step 101: Determine the video features corresponding to the video to be detected and the probability information of the video to be detected.

[0034] Among them, the video to be detected includes plant growth information, and the probability information is used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected.

[0035] The plants in this embodiment can be crops, ornamental plants, wild plants, etc. Exemplarily, the plants in this embodiment can be, for example: tomatoes, peppers, wheat, corn, etc. The video to be detected in this embodiment refers to the video formed by shooting the growth process of plants.

[0036] The main purpose of the plant growth stage detection method provided in this embodiment is to locate and identify the growth process of plants (such as leaf growth, flowering, fruiting, etc.), locate when an action occurs, and then identify the category to which each action belongs. Taking the important agricultural crop - pepper as an example, the growth stage of pepper includes four parts: germination, growth, flowering, and fruiting. Peppers have different fertilizer requirements during different growth periods. From flowering to fruit setting, more nitrogen fertilizer is needed; from fruit setting to maturity, more potassium fertilizer is needed. Accurately detecting the growth stage of peppers and applying fertilizers precisely can significantly increase the pepper yield.

[0037] The target object of action detection in the related art is a person, the action boundaries are relatively clear, and the correlation between actions is not high. However, plant growth is a continuous and uninterrupted process, and the morphology, color, and cycle of different plants during the growth process are also different. Although the goals of the two are the same, using the classic action detection scheme to identify the boundaries of plant growth, there are problems such as inaccurate boundaries of the plant growth process and even the inversion of the growth process in the detection results.

[0038] To solve the above problems, in step 101, the probability information of the video to be detected can be determined. The probability information in this embodiment can characterize the action boundary information in the video to be detected, and it is specifically used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected. The boundary probability in this embodiment can include the start probability and the end probability. The start probability in this embodiment refers to the probability that an action starts at a certain time point. The end probability in this embodiment refers to the probability that an action ends at a certain time point. The action probability in this embodiment refers to the probability that an action occurs at a certain time point. The time in this embodiment refers to the duration of the video to be detected. The time point in this embodiment can also be referred to as the time position.

[0039] In one implementation, video features corresponding to the video to be detected and probability information of the video to be detected can be obtained based on existing video processing algorithms. For example, video features corresponding to the video to be detected can be obtained based on methods such as scale-invariant feature transformation and oriented gradient histogram. For example, probability information of the video to be detected can be obtained based on existing time series detection methods and action recognition methods.

[0040] In another implementation, the video features corresponding to the video to be detected and the probability information of the video to be detected may be determined based on the following steps 1011 and 1012.

[0041] Step 1011: Input each image frame of the video to be detected and the stacked optical flow field of the image frames into the first neural network to obtain the video features output by the first neural network.

[0042] The video features include features corresponding to each image frame. The first neural network is used to encode the image frames and the stacked optical flow fields of the image frames and then splice them to obtain the features corresponding to each image frame.

[0043] Figure 2 FIG. 1 is a schematic diagram of determining video features and probability information in an embodiment of the present invention. Figure 2 As shown, the first neural network in this embodiment is a neural network with a dual-stream structure. The first neural network processes image frames (i.e. Figure 2 The first neural network performs feature encoding on the image frame through a spatial network to obtain features of the image frame. The first neural network performs feature encoding on the stacked optical flow of the image frame through a temporal network to obtain features of the stacked optical flow field. Afterwards, the first neural network fuses (for example, splices) the features of the image frame and the features of the stacked optical flow field to obtain features corresponding to each image frame. Exemplarily, the image frame in this embodiment may be a red, green, and blue (RGB) frame.

[0044] The video to be detected in this embodiment can be expressed as l v is the number of image frames included in the video to be detected, which can represent the length of the video to be detected. It consists of two parts: is the tth n image frames, It is centered around The stacked optical flow field is used as the center frame. The first neural network extracts spatiotemporal features through the convolution kernel included in the spatial network and the temporal network, and then undergoes multi-layer convolution and pooling ( Figure 2 After that, the two-stream features are fused and then global average pooling ( Figure 2(not shown in the figure), output the video features corresponding to the video to be detected l s refers to the number of image frames obtained by sampling the video to be detected at intervals of σ. It can be understood that l s = l v / σ. Of course, σ can also be taken as 1. In this case, l s = l v .

[0045] The stacked optical flow field in this embodiment refers to stacking the motion information (optical flow) between consecutive image frames into a multi-channel tensor in chronological order. This method can capture the motion information in the video, thereby providing rich spatio-temporal features for video analysis and processing.

[0046] The video features obtained based on the first neural network include temporal features and spatial features, which can further improve the accuracy of subsequent detection results.

[0047] Step 1012: Input the video features into the second neural network to obtain the probability information output by the second neural network.

[0048] Among them, the second neural network is used to determine the probability information according to the temporal convolutional module it includes.

[0049] In step 1012, the video features extracted by the first neural network are used as the input of the second neural network. In the second neural network, the temporal modeling ability of the temporal convolutional network it includes is used to capture the local semantic information of the video. For example, the information of the boundary probability and the action probability changing with time.

[0050] Optionally, the temporal convolutional network of the second neural network in this embodiment can have three layers, namely Conv(512, 3, Relu), Conv(512, 3, Relu), and Conv(3, 1, Sigmoid). Among them, Conv(c f , c k , Act) where c f represents the number of convolutional kernels of the temporal convolutional layer, c k represents the size of the temporal convolutional layer, and Act represents the activation function of the temporal convolutional layer. The time steps of the three temporal convolutional layers are the same, all being 1. In the last temporal convolutional layer, the mapping relationships between the boundary probability and time, and the action probability and time are generated respectively through the Sigmoid classifier. Further, in the last temporal convolutional layer, three Sigmoid classifiers are used to generate three probability sequences Among them, is the start probability at time position t n , Time position t n The end probability of is the action probability of time position t n .

[0051] In step 1012, based on the video features including spatio-temporal features, the probability information corresponding to the video to be detected is obtained, which improves the accuracy of the probability information. Thus, the accuracy of the detection result is further improved. Moreover, by fusing local and global features through temporal convolutional operations, the context information near the boundary is captured to obtain the probability information, enhancing the ability to judge fuzzy boundaries, avoiding the limitations of heuristic methods such as sliding windows, and significantly improving the positioning accuracy of start and end points.

[0052] Step 102: Determine the candidate time intervals in the video to be detected according to the probability information.

[0053] Among them, the confidence level of the candidate time interval is greater than the preset confidence threshold. The confidence level of the candidate time interval is a value determined according to the corresponding boundary probability and action probability.

[0054] In this embodiment, the probability information is used to locate the start and end points of the candidate action segments. Therefore, the candidate time intervals in the video to be detected can be determined according to the probability information.

[0055] In one implementation, first determine the first time point and the second time point corresponding to the boundary probability greater than the preset boundary probability threshold, and determine the third time point corresponding to the action probability greater than the preset action probability threshold. Combine the first time point and the second time point to obtain a time interval. According to the action probability corresponding to the third time point, the boundary probability corresponding to the first time point, and the boundary probability corresponding to the second time point in the time interval, determine the confidence level of each time interval. Determine the time intervals with a confidence level greater than the preset confidence threshold as the candidate time intervals.

[0056] In another implementation, the boundary probability includes the start probability and the end probability. The probability information includes: the start probability sequence, the end probability sequence, and the action probability sequence. Determine the preliminary screening start time points as the time points in the start probability sequence where the corresponding probability is higher than the first probability threshold and is a local extremely high probability. Determine the preliminary screening end time points as the time points in the end probability sequence where the corresponding probability is higher than the second probability threshold and is a local extremely high probability. Form each preliminary screening time interval according to the preliminary screening start time points and the preliminary screening end time points. Determine the confidence level of the preliminary screening time interval according to the start probability, end probability, and action probability corresponding to the time points within the preliminary screening time interval. Determine the preliminary screening time intervals with a corresponding confidence level greater than the preset confidence threshold as the candidate time intervals. This implementation will be described in detail in the subsequent embodiments.

[0057] Exemplarily, assume that the duration of the video to be detected in this embodiment is 10 minutes. Then, the candidate time intervals determined in step 102 can be (2 minutes, 3 minutes), (4 minutes, 6 minutes), and (8 minutes, 10 minutes).

[0058] Step 103: Input the video features, probability information, and candidate time intervals into the target neural network to obtain the detection result output by the target neural network.

[0059] Among them, the target neural network is used to determine the detection result by combining the prior information of plant growth. The detection result includes: at least one target time interval and the corresponding plant growth stage for each target time interval.

[0060] The target neural network in this embodiment can determine the detection result according to the video features, probability information, candidate time intervals, and prior information of plant growth.

[0061] Since the plant growth process (germination, growth, flowering, fruiting) is a continuous and complete process, prior knowledge of plant growth can be used to determine the prior information of plant growth. Establish an explicit mapping relationship between botanical knowledge and the target neural network to solve the misjudgment problem caused by ignoring the plant development law in traditional time series detection algorithms, and further eliminate the time series intervals that do not conform to the plant development law. The core lies in transforming the biological rules of plant growth into computable mathematical constraints.

[0062] Assume that the plant growth stage is divided into 4 stages: 1. Germination stage; 2. Leaf growth stage; 3. Flowering stage; 4. Fruiting stage.

[0063] According to the biological irreversibility, there are the following hard constraint rules: Reverse transfer between plant growth stages is prohibited. For example, the flowering stage cannot return to the leaf growth stage; self-looping is allowed within the growth stage. For example, the leaf growth stage can maintain its own state, and the next stage can also be the leaf growth stage; the fruiting stage can only transfer to itself. The fruiting stage is the last stage of the last plant growth stage and cannot be transferred to other stages.

[0064] At the same time, there are the following soft constraint rules: The transition from leaf growth to flowering is related to the Flowering Biological Accumulated Time Unit (FBDTU). FBDTU is a time parameter. When the time is appropriate, the transition probability is relatively high.

[0065] According to the above hard constraint rules and soft constraint rules, during the training process of the target neural network, the prior information of plant growth can be generated. The target neural network combines the prior information of plant growth and the input video features, probability information, and candidate time intervals to jointly determine the detection result.

[0066] For example, in this embodiment, the detection results may include: (2 minutes, 5 minutes), germination; (8 minutes, 10 minutes), growth; (12 minutes, 13 minutes), flowering.

[0067] In the method for detecting plant growth stages provided in this embodiment, the probability information may represent the action boundary information in the video to be detected, and the plant growth prior information may represent the prior knowledge of plant growth. The confidence level of the candidate time interval is greater than the preset confidence threshold. Therefore, the detection results determined by the target neural network in combination with the plant growth prior information, the input video features, the probability information, and the candidate time interval can effectively alleviate the problem of fuzzy plant growth boundaries. At the same time, it solves the misjudgment problem caused by the detection algorithm in the related art ignoring the plant development law.

[0068] It should be noted that the time interval (t1, t2) in this embodiment or subsequent embodiments can be understood as [t1, t2) or (t1, t2].

[0069] The technical solution of the embodiment of the present invention includes: determining the video features corresponding to the video to be detected and the probability information of the video to be detected, where the video to be detected includes plant growth information, and the probability information is used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected; determining the candidate time interval in the video to be detected according to the probability information, where the confidence level of the candidate time interval is greater than the preset confidence threshold, and the confidence level of the candidate time interval is a value determined according to the corresponding boundary probability and action probability; inputting the video features, the probability information, and the candidate time interval into the target neural network to obtain the detection results output by the target neural network, where the target neural network is used to determine the detection results in combination with the plant growth prior information, and the detection results include: at least one target time interval and the plant growth stage corresponding to each target time interval. In this method for detecting plant growth stages, the target neural network can comprehensively determine the detection results based on the video features, the probability information, the candidate time interval, and the plant growth prior information. The probability information can represent the action boundary information in the video to be detected, effectively alleviating the impact of fuzzy plant growth boundaries on the detection results. The plant growth prior information can represent the prior knowledge of plant growth, solving the misjudgment problem caused by the detection algorithm in the related art ignoring the plant development law. Therefore, the accuracy of the detection results of the method for detecting plant growth stages provided in this embodiment is relatively high.

[0070] Figure 3 is a flowchart of another method for detecting plant growth stages provided by the embodiment of the present invention. The method for detecting plant growth stages provided in this embodiment is in Figure 1Based on the illustrated embodiments and various alternative implementations, the implementation manner of how to determine the candidate time interval will be described in detail. For simplicity, only the specific implementation manner of step 102 will be described in this embodiment. In this embodiment, the boundary probability includes the start probability and the end probability, and the probability information includes: the start probability sequence, the end probability sequence, and the action probability sequence. As Figure 3 shown, step 102 in the plant growth stage detection method provided in this embodiment includes the following steps 1021 to 1025.

[0071] Step 1021: Determine the preliminary screening start time point as the time point in the start probability sequence where the corresponding start probability is higher than the first probability threshold and is a local extremely high probability.

[0072] The local extremely high probability in step 1021 represents the inflection point probability among the start probabilities higher than the first probability threshold in the start probability sequence.

[0073] The reason for restricting the start probability to be higher than the first probability threshold is to avoid the problem of low calculation efficiency caused by determining the preliminary screening start time point as the time point corresponding to a relatively small but local extremely high start probability.

[0074] Step 1022: Determine the preliminary screening end time point as the time point in the end probability sequence where the corresponding end probability is higher than the second probability threshold and is a local extremely high probability.

[0075] The local extremely high probability in step 1022 represents the inflection point probability among the end probabilities higher than the second probability threshold in the end probability sequence.

[0076] The reason for restricting the end probability to be higher than the second probability threshold is to avoid the problem of low calculation efficiency caused by determining the preliminary screening end time point as the time point corresponding to a relatively small but local extremely high end probability.

[0077] In this embodiment, the start probability is used to locate the starting point of the candidate action segment, and the end probability is used to locate the ending point of the candidate action segment. Therefore, based on the information of the boundary probability changing with time, the preliminary screening start time point and the preliminary screening end time point can be determined. And the action probability is used to verify the action confidence within the following preliminary screening time interval and filter out background or noise segments.

[0078] Step 1023: Determine each preliminary screening time interval according to the preliminary screening start time point and the preliminary screening end time point.

[0079] Among them, each preliminary screening time interval includes the preliminary screening start time point and the preliminary screening end time point, and the preliminary screening end time point is greater than the preliminary screening start time point.

[0080] Assume the preliminary screening start time point is ts The initial screening end time point is t e Combine all t that satisfy t e >t s of t s and t e to obtain each initial screening time interval.

[0081] For example, assume the initial screening start time points are 1 minute, 3 minutes, 5 minutes, and 6 minutes, and the initial screening end time points are 2 minutes, 4 minutes, and 7 minutes. Then the determined initial screening time intervals include: (1 minute, 2 minutes), (1 minute, 4 minutes), (1 minute, 7 minutes), (3 minutes, 4 minutes), (3 minutes, 7 minutes), (5 minutes, 7 minutes), and (6 minutes, 7 minutes).

[0082] Step 1024: For each initial screening time interval, determine the confidence of the initial screening time interval according to the duration of the initial screening time interval, the start probability, end probability, and action probability corresponding to the time points within the initial screening time interval.

[0083] Exemplarily, the confidence of each initial screening time interval can be determined according to the formula wherein, represents the action probability of a certain time point in the initial screening time interval, represents the start probability of a certain time point in the initial screening time interval, represents the end probability of a certain time point in the initial screening time interval.

[0084] Step 1025: Determine the initial screening time intervals with corresponding confidence greater than the preset confidence threshold as candidate time intervals.

[0085] In the above implementation method, strengthening the modeling of complete action segments can efficiently generate compact and high-precision candidate time intervals, providing reliable input for subsequent classification and positioning.

[0086] Furthermore, step 1025 includes the following steps 10251 to 10253.

[0087] Step 10251: If the initial screening time intervals with corresponding confidence greater than the preset confidence threshold do not include low-confidence time intervals, then determine the initial screening time interval as a candidate time interval.

[0088] wherein, the low-confidence time interval is the initial screening time interval with corresponding confidence less than the preset confidence threshold.

[0089] Step 10252: If at least a part of the low-confidence time intervals are included in the preliminary screening time interval where the corresponding confidence level is greater than the preset confidence threshold, and the overlap degree between the low-confidence time interval and the preliminary screening time interval is greater than the preset overlap threshold, then remove the time interval overlapping with the low-confidence time interval from the preliminary screening time interval, and determine the remaining time interval of the preliminary screening time interval as the candidate time interval.

[0090] In step 10252, in order to further improve the accuracy of the determined candidate time interval, when at least a part of the low-confidence time intervals are included in the preliminary screening time interval where the corresponding confidence level is greater than the preset confidence threshold, and the overlap degree between the low-confidence time interval and the preliminary screening time interval is greater than the preset overlap threshold, remove the time interval overlapping with the low-confidence time interval from the preliminary screening time interval, and determine the remaining time interval of the preliminary screening time interval as the candidate time interval.

[0091] For example, assume that the preliminary screening time interval where the corresponding confidence level is greater than the preset confidence threshold is (1 minute, 5 minutes), and the low-confidence time interval is (3 minutes, 6 minutes). Assume that the preset overlap threshold is 30%. Then remove the time interval overlapping with (3 minutes, 6 minutes) from the preliminary screening time interval (1 minute, 5 minutes), and determine the remaining time interval (1 minute, 3 minutes) of (1 minute, 5 minutes) as the candidate time interval.

[0092] Step 10253: If at least a part of the low-confidence time intervals are included in the preliminary screening time interval where the corresponding confidence level is greater than the preset confidence threshold, and the overlap degree between the low-confidence time interval and the preliminary screening time interval is less than or equal to the preset overlap threshold, then determine the preliminary screening time interval as the candidate time interval.

[0093] Steps 10251 to 10253 can further remove the time interval overlapping with the low-confidence time interval from the preliminary screening time interval where the corresponding confidence level is greater than the preset confidence threshold on the premise of meeting the conditions, thereby further improving the accuracy of the candidate time interval.

[0094] The plant growth stage detection method provided in this embodiment determines the confidence level of each preliminary screening time interval according to the duration of the preliminary screening time interval, the start probability, the end probability, and the action probability corresponding to the time points within the preliminary screening time interval. The preliminary screening time interval with the corresponding confidence level greater than the preset confidence threshold is determined as the candidate time interval, realizing the efficient generation of high-precision candidate time intervals, thereby further improving the accuracy of the detection result and the detection efficiency.

[0095] Figure 4It is a flowchart of yet another plant growth stage detection method provided by an embodiment of the present invention. The plant growth stage detection method provided in this embodiment, on the basis of the embodiments and various optional implementation manners shown in Figure 1 or Figure 3 , details the implementation manner of how to obtain the detection result output by the target neural network. For simplicity, only the specific implementation manner of step 103 is described in this embodiment. The target neural network in this embodiment includes a boundary attention module, an encoder module, a decoder module, and a morphological prior constraint module connected in sequence. As shown in Figure 4 , step 103 in the plant growth stage detection method provided in this embodiment includes the following step 1031.

[0096] Step 1031: Input the video feature and probability information into the boundary attention module, and input the candidate time interval into the decoder module to obtain the detection result output by the morphological prior constraint module.

[0097] Among them, the morphological prior constraint module is used to verify the result output by the decoder module according to the plant growth prior information to obtain the detection result.

[0098] Figure 5 It is a schematic structural diagram of the target neural network in an embodiment of the present invention. As shown in Figure 5 , the target neural network includes a boundary attention module, an encoder module, a decoder module, and a morphological prior constraint module connected in sequence.

[0099] Optionally, the boundary attention module in this embodiment is used to obtain the video feature including the action boundary according to the video feature, start probability, end probability, and action probability. Specifically, the boundary attention module can multiply the video feature, start probability, end probability, and action probability.

[0100] The video feature including the action boundary is encoded by the encoder module and then input into the decoder module together with the candidate time interval. The decoder in this embodiment can be a relaxation decoder and can implement the self-attention mechanism. The decoder in this embodiment can pay attention to a wider time context during decoding instead of strictly adjacent regions, so as to capture the start and end ambiguity of the action, and then establish the relationship between different candidate time intervals.

[0101] Optionally, the output layer of the decoder module in this embodiment can include a multi-branch detection head. The result output by the multi-branch detection head is verified by the morphological prior constraint module to obtain the detection result.

[0102] Optionally, the multi-branch detection head in this embodiment may include a boundary branch, a completeness branch, and an action classification branch. Among them, the boundary branch directly regresses the start time and end time of the action through a fully connected layer, that is, outputs a time interval. The completeness branch uses a Sigmoid classifier to output an action completeness score p conf , p conf is the overlap degree between the temporal action interval and the ground truth. The action classification branch jointly predicts the action category and outputs a confidence score for the time interval, indicating the probability that the interval contains the action. The morphological prior constraint module is used to verify the result output by the decoder module according to the prior information of plant growth to obtain the detection result, so as to improve the accuracy of the detection result.

[0103] The method for detecting plant growth stages based on the decoder module in the related art has the problem of over-smoothing, which reduces the ability to distinguish action boundaries, and the growth stage boundaries become blurred.

[0104] The method for detecting plant growth stages provided in this embodiment can input video features and probability information into the boundary attention module, input the candidate time interval into the decoder module, and obtain the detection result output by the morphological prior constraint module. Among them, the morphological prior constraint module is used to verify the result output by the decoder module according to the prior information of plant growth to obtain the detection result. Based on this neural network structure, a method for detecting plant growth stages based on the boundary attention module is provided, which realizes efficient and accurate output of detection results and further improves the accuracy of the detection results. This detection method can accurately detect the plant growth stages of different species, different growth stages, with large variations in the time span of growth stages, and continuous growth stage times.

[0105] In one embodiment, the results output by the decoder module include: each secondary screening time interval, the corresponding plant growth stage of each secondary screening time interval, and the confidence of each secondary screening time interval. At least one secondary screening time interval corresponds to at least two plant growth stages.

[0106] The step of "obtaining the detection result output by the morphological prior constraint module" in step 1031 includes the following steps 10311 to 10316.

[0107] Step 10311: Through the morphological prior constraint module, select a plant growth stage for the secondary screening time interval corresponding to at least two plant growth stages, and sort each secondary screening time interval in chronological order to obtain an initial time interval sequence.

[0108] For example, assume that the secondary screening time intervals include (t1, t2), (t3, t4), and (t5, t6). (t3, t4) corresponds to the plant growth stage: stage 3, and (t5, t6) corresponds to the plant growth stage: stage 4. Assume that the secondary screening time interval (t1, t2) corresponds to two plant growth stages: stage 1 and stage 2. In step 10311, first select stage 1 for the secondary screening time interval (t1, t2). Then, sort these three secondary screening time intervals in chronological order to obtain an initial time interval sequence. Sorting in chronological order can be sorting in ascending order of the start time points in the secondary screening time intervals. Assume t1 < t3 < t5, then the initial time interval sequence is obtained: (t1, t2), (t3, t4), (t5, t6).

[0109] Step 10312: Through the morphological prior constraint module, traverse each pair of adjacent secondary screening time intervals in the initial time interval sequence, and eliminate the secondary screening time intervals that do not conform to the plant growth prior information to obtain a screened time interval sequence.

[0110] The plant growth prior information in this embodiment may include a plant growth prior probability matrix. The plant growth prior probability matrix is an m*m matrix, where m represents the number of plant growth stages. The element a in the plant growth prior probability matrix ij represents the probability from stage i to stage j. Exemplarily, m can be 4.

[0111] When training the plant growth prior information, illegal transitions can be prohibited according to biological rules, and zeros are set at the corresponding positions in the matrix. For example, the probability from the flowering stage to the leaf-growing stage is 0. Other positions can be assigned initial probability values using the Sigmoid function, and finally, each row is normalized. This matrix is embedded as a trainable weight into the target neural network. The element M{ij} in the plant growth prior probability matrix = Softmax(W{ij} * φ(x_t) + b_{ij}). Where φ(x_t) represents the initial probability value, and W_{ij} and b_{ij} are learnable parameters. The KL divergence loss (Kullback-Leibler Divergence Loss) is used to ensure that the trained plant growth prior probability matrix does not deviate from the biological prior knowledge (i.e., the above hard rules and soft rules).

[0112] In step 10312, traverse each pair of adjacent secondary screening time intervals in the initial time interval sequence, and eliminate the secondary screening time intervals that do not conform to the plant growth prior information to obtain a screened time interval sequence.

[0113] Step 10313: Through the morphological prior constraint module, according to the transition probability between each pair of adjacent second-screening time intervals in the filtered time interval sequence and the confidence of the adjacent second-screening time intervals in the prior information of plant growth, determine the rationality score of the filtered time interval sequence.

[0114] Assume that the filtered time interval sequence is (t1, t2), (t3, t4), (t5, t6). Then, for stage 1 corresponding to (t1, t2) and stage 3 corresponding to (t3, t4), determine the transition probability A1 from stage 1 to stage 3 from the prior information of plant growth. For stage 3 corresponding to (t3, t4) and stage 4 corresponding to (t5, t6), determine the transition probability A2 from stage 1 to stage 3 from the prior information of plant growth.

[0115] After that, based on the transition probability A1, the transition probability A2, and the confidence of (t1, t2), (t3, t4), (t5, t6), determine the rationality score of the filtered time interval sequence. Exemplarily, the rationality score of the filtered time interval sequence can be determined in the following way: Determine the product B1 of the transition probability A1 and the sum of the confidence of (t1, t2) and the confidence of (t3, t4); Determine the product B2 of the transition probability A2 and the sum of the confidence of (t3, t4) and the confidence of (t5, t6); Determine the sum of B1 and B2 as the rationality score of the filtered time interval sequence.

[0116] Step 10314: If there are unselected plant growth stages in the second-screening time intervals corresponding to at least two plant growth stages, then through the morphological prior constraint module, return to execute the steps of Step 10311.

[0117] Continuing with the above example, there is an unselected plant growth stage in the second-screening time interval (t1, t2): stage 2, then return to execute the steps of Step 10311.

[0118] Step 10315: If there are no unselected plant growth stages in the second-screening time intervals corresponding to at least two plant growth stages, then stop the iteration.

[0119] Step 10316: Through the morphological prior constraint module, determine the second-screening time intervals included in the filtered time interval sequence with the highest corresponding rationality score as the target time intervals, and determine the plant growth stages corresponding to the second-screening time intervals included in the filtered time interval sequence as the plant growth stages corresponding to the target time intervals.

[0120] The above steps 10311 to 10316 can be implemented, which can realize the traversal of the plant growth stages in the secondary screening time intervals corresponding to at least two plant growth stages. Furthermore, the rationality scores corresponding to the screened time interval sequences can be obtained. The secondary screening time intervals included in the screened time interval sequence with the highest corresponding rationality score are determined as the target time intervals, and the plant growth stages corresponding to the secondary screening time intervals included in the screened time interval sequence are determined as the plant growth stages corresponding to the target time intervals. Therefore, the morphological prior constraint module can comprehensively, accurately, and efficiently verify the results output by the decoder, further improving the accuracy of the detection results and the detection efficiency.

[0121] Furthermore, the detection results also include the confidence levels corresponding to each target time interval. The confidence level corresponding to the target time interval in this embodiment can be the confidence level output by the decoder module. After step 103, the plant growth stage detection method provided in this embodiment further includes the following steps 104 to 108.

[0122] Step 104: Through the morphological prior constraint module, sort each target time interval in chronological order to obtain a target time interval sequence.

[0123] The chronological order here is similar to the aforementioned chronological order and will not be elaborated here.

[0124] Step 105: If there is an overlap between two adjacent target time intervals in the target time interval sequence, then through the morphological prior constraint module, traverse all splitting schemes to obtain at least two updated time interval sequences.

[0125] Among them, the updated time interval sequence includes: non-overlapping target time intervals and splitting time intervals. The splitting scheme is used to indicate the start time point and end time point of the splitting time interval, as well as the plant growth stage and confidence level corresponding to the splitting time interval.

[0126] For example, if the target time intervals (2 minutes, 5 minutes) and (4 minutes, 6 minutes) overlap, the splitting schemes can include: (2 minutes, 5 minutes) remains unchanged, (4 minutes, 6 minutes) is split into (5 minutes, 6 minutes), and the plant growth stage of (5 minutes, 6 minutes) is the plant growth stage corresponding to (4 minutes, 6 minutes), and the confidence level of (5 minutes, 6 minutes) is half of the confidence level of (4 minutes, 6 minutes); (4 minutes, 6 minutes) remains unchanged, (2 minutes, 5 minutes) is split into (2 minutes, 4 minutes), and the plant growth stage of (2 minutes, 4 minutes) is the plant growth stage corresponding to (2 minutes, 5 minutes), and the confidence level of (2 minutes, 4 minutes) is two-thirds of the confidence level of (2 minutes, 5 minutes);..., etc.

[0127] It should be noted that in the updated time interval sequence obtained based on the splitting scheme: (2 minutes, 5 minutes) remains unchanged, and (4 minutes, 6 minutes) is split into (5 minutes, 6 minutes): (2 minutes, 5 minutes), (5 minutes, 6 minutes), (2 minutes, 5 minutes) is the target time interval, and (5 minutes, 6 minutes) is the splitting time interval.

[0128] In step 105, all splitting schemes are traversed to obtain at least two updated time interval sequences.

[0129] If there is no overlap between two adjacent target time intervals in the target time interval sequence, step 106 is directly executed.

[0130] Step 106: For each updated time interval sequence, through the morphological prior constraint module, all completion schemes are traversed to obtain at least two secondary updated time interval sequences.

[0131] Among them, the secondary updated time interval sequence includes: target time intervals, splitting time intervals, and completion time intervals that do not overlap with each other and there is no time interval between adjacent time intervals. The completion scheme is used to indicate the plant growth stage and confidence level corresponding to the completion time interval.

[0132] Suppose the updated time interval sequence is: (1 minute, 3 minutes), (5 minutes, 6 minutes). There is a time interval between 3 minutes and 5 minutes. The completion schemes in this example can include: the plant growth stage corresponding to the completion time interval (3 minutes, 5 minutes) is the same as the plant growth stage corresponding to (1 minute, 3 minutes), and the confidence level of the completion time interval (3 minutes, 5 minutes) is the confidence level of (1 minute, 3 minutes); the plant growth stage corresponding to the completion time interval (3 minutes, 5 minutes) is the same as the plant growth stage corresponding to (5 minutes, 6 minutes), and the confidence level of the completion time interval (3 minutes, 5 minutes) is the confidence level of (5 minutes, 6 minutes);..., etc.

[0133] In step 106, for each updated time interval sequence, through the morphological prior constraint module, all completion schemes are traversed to obtain at least two secondary updated time interval sequences.

[0134] Step 107: For each secondary updated time interval sequence, through the morphological prior constraint module, according to the transition probability between each pair of adjacent time intervals in the secondary updated time interval sequence and the confidence levels of the adjacent time intervals in the plant growth prior information, the rationality score of the secondary updated time interval sequence is determined.

[0135] The implementation process and technical principle of step 107 are similar to those of step 10313, and will not be elaborated here.

[0136] It can be understood that each pair of adjacent time intervals in step 107 may include at least one of the following: a target time interval, a split time interval, and a complement time interval.

[0137] Step 108: Through the morphological prior constraint module, determine the time intervals included in the sequence of secondary updated time intervals with the highest corresponding rationality score as the final time intervals, and determine the plant growth stage corresponding to the time intervals included in the sequence of secondary updated time intervals as the plant growth stage corresponding to the final time intervals.

[0138] Based on the final time intervals obtained from the above steps 104 to 108, the time axis of the video to be detected can be covered, with time continuity, realizing a comprehensive detection of the video to be detected, and being more in line with actual requirements.

[0139] In this embodiment, through the above steps 104 to 108, a comprehensive detection within the time range of the video to be detected can be efficiently realized, making the final detection result more in line with actual requirements.

[0140] Figure 6 It is a schematic structural diagram of a plant growth stage detection device provided by an embodiment of the present invention. As Figure 6 shown, the plant growth stage detection device provided in this embodiment includes the following modules: a first determination module 61, a second determination module 62, and a third determination module 63. <*

[0141] The first determination module 61 is configured to determine the video features corresponding to the video to be detected and the probability information of the video to be detected.

[0142] Among them, the video to be detected includes plant growth information. The probability information is used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected.

[0143] The second determination module 62 is configured to determine candidate time intervals in the video to be detected according to the probability information.

[0144] Among them, the confidence level of the candidate time interval is greater than a preset confidence threshold. The confidence level of the candidate time interval is a value determined according to the corresponding boundary probability and action probability.

[0145] The third determination module 63 is configured to input the video features, the probability information, and the candidate time intervals into a target neural network, and obtain the detection result output by the target neural network.

[0146] Among them, the target neural network is used to determine the detection result in combination with plant growth prior information. The detection result includes: at least one target time interval and the plant growth stage corresponding to each target time interval.

[0147] In one embodiment, the first determination module 61 is specifically configured to: input each image frame of the video to be detected and the stacked optical flow field of the image frame into a first neural network to obtain video features output by the first neural network, where the video features include features corresponding to each image frame, and the first neural network is used to encode the image frame and the stacked optical flow field of the image frame respectively and then splice them to obtain features corresponding to each image frame; input the video features into a second neural network to obtain probability information output by the second neural network, where the second neural network is used to determine the probability information according to the temporal convolutional module included therein.

[0148] In one embodiment, the boundary probability includes a start probability and an end probability, and the probability information includes: a start probability sequence, an end probability sequence, and an action probability sequence. The second determination module 62 is specifically configured to: determine, as a preliminary screening start time point, a time point in the start probability sequence where the corresponding start probability is higher than a first probability threshold and is a local extremely high probability; determine, as a preliminary screening end time point, a time point in the end probability sequence where the corresponding end probability is higher than a second probability threshold and is a local extremely high probability; determine respective preliminary screening time intervals according to the preliminary screening start time point and the preliminary screening end time point, where each preliminary screening time interval includes a preliminary screening start time point and a preliminary screening end time point, and the preliminary screening end time point is greater than the preliminary screening start time point; for each preliminary screening time interval, determine the confidence of the preliminary screening time interval according to the duration of the preliminary screening time interval, the start probability, the end probability, and the action probability corresponding to the time points within the preliminary screening time interval; and determine the preliminary screening time interval with a corresponding confidence greater than a preset confidence threshold as the candidate time interval.

[0149] In one embodiment, in the aspect of determining the initial screening time interval with a corresponding confidence level greater than the preset confidence threshold as the candidate time interval, the second determination module 62 is specifically configured to: if the initial screening time interval with a corresponding confidence level greater than the preset confidence threshold does not include a low-confidence time interval, determine the initial screening time interval as the candidate time interval, where the low-confidence time interval is an initial screening time interval with a corresponding confidence level less than the preset confidence threshold; if the initial screening time interval with a corresponding confidence level greater than the preset confidence threshold includes at least a part of the low-confidence time interval, and the overlap degree between the low-confidence time interval and the initial screening time interval is greater than the preset overlap threshold, remove the time interval overlapping with the low-confidence time interval from the initial screening time interval, and determine the remaining time interval of the initial screening time interval as the candidate time interval; if the initial screening time interval with a corresponding confidence level greater than the preset confidence threshold includes at least a part of the low-confidence time interval, and the overlap degree between the low-confidence time interval and the initial screening time interval is less than or equal to the preset overlap threshold, determine the initial screening time interval as the candidate time interval.

[0150] In one embodiment, the target neural network includes a boundary attention module, an encoder module, a decoder module, and a morphological prior constraint module connected in sequence. The third determination module 63 is specifically configured to: input the video feature and the probability information into the boundary attention module, input the candidate time interval into the decoder module, and obtain the detection result output by the morphological prior constraint module, where the morphological prior constraint module is used to verify the result output by the decoder module according to the plant growth prior information to obtain the detection result.

[0151] In one embodiment, the results output by the decoder module include: each secondary screening time interval, the plant growth stage corresponding to each secondary screening time interval, and the confidence level of each secondary screening time interval. At least one secondary screening time interval corresponds to at least two plant growth stages. In terms of obtaining the detection results output by the morphological prior constraint module, the third determination module 63 is specifically configured to: through the morphological prior constraint module, select a plant growth stage for the secondary screening time intervals corresponding to at least two plant growth stages, sort each secondary screening time interval in chronological order to obtain an initial time interval sequence; through the morphological prior constraint module, traverse each pair of adjacent secondary screening time intervals in the initial time interval sequence, and eliminate the secondary screening time intervals that do not conform to the plant growth prior information to obtain a screened time interval sequence; through the morphological prior constraint module, according to the transition probability of each pair of adjacent secondary screening time intervals in the screened time interval sequence in the plant growth prior information, and the confidence level of the adjacent secondary screening time intervals, determine the rationality score of the screened time interval sequence; if there are unselected plant growth stages for the secondary screening time intervals corresponding to at least two plant growth stages, then through the morphological prior constraint module, return to execute the step of "through the morphological prior constraint module, select a plant growth stage for the secondary screening time intervals corresponding to at least two plant growth stages, sort each secondary screening time interval in chronological order to obtain an initial time interval sequence"; if there are no unselected plant growth stages for the secondary screening time intervals corresponding to at least two plant growth stages, then stop the iteration; through the morphological prior constraint module, determine the secondary screening time intervals included in the screened time interval sequence with the highest corresponding rationality score as the target time intervals, and determine the plant growth stages corresponding to the secondary screening time intervals included in the screened time interval sequence as the plant growth stages corresponding to the target time intervals.

[0152] In one embodiment, the detection results further include the confidence levels corresponding to each target time interval. The apparatus further includes: a sorting module, a fourth determination module, a fifth determination module, a sixth determination module, and a seventh determination module.

[0153] The sorting module is configured to, through the morphological prior constraint module, sort each target time interval in chronological order to obtain a target time interval sequence.

[0154] A fourth determination module, configured to, if there is an overlap between two adjacent target time intervals in the target time interval sequence, traverse all splitting schemes through the morphological prior constraint module to obtain at least two updated time interval sequences. Wherein, the updated time interval sequence includes: non-overlapping target time intervals and splitting time intervals, and the splitting scheme is used to indicate the start time point and end time point of the splitting time interval, and to indicate the plant growth stage and confidence level corresponding to the splitting time interval.

[0155] A fifth determination module, configured to, for each of the updated time interval sequences, traverse all completion schemes through the morphological prior constraint module to obtain at least two secondarily updated time interval sequences. Wherein, the secondarily updated time interval sequence includes: non-overlapping target time intervals, splitting time intervals, and completion time intervals with no time gap between adjacent time intervals. The completion scheme is used to indicate the plant growth stage and confidence level corresponding to the completion time interval.

[0156] A sixth determination module, configured to, for each of the secondarily updated time interval sequences, through the morphological prior constraint module, determine a rationality score of the secondarily updated time interval sequence according to the transition probability between each pair of adjacent time intervals in the secondarily updated time interval sequence and the confidence levels of the adjacent time intervals in the plant growth prior information.

[0157] A seventh determination module, configured to, through the morphological prior constraint module, determine the time intervals included in the secondarily updated time interval sequence with the highest corresponding rationality score as the final time intervals, and determine the plant growth stage corresponding to the time intervals included in the secondarily updated time interval sequence as the plant growth stage corresponding to the final time intervals.

[0158] The plant growth stage detection device provided by the embodiments of the present invention can execute the plant growth stage detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0159] Figure 7 It is a schematic structural diagram of an electronic device for implementing the plant growth stage detection method of the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0160] AsFigure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0161] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0162] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the plant growth stage detection method.

[0163] In some embodiments, the plant growth stage detection method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the plant growth stage detection method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the plant growth stage detection method by any other suitable means (e.g., by means of firmware).

[0164] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0165] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other device, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0166] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0168] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0169] A computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0170] An embodiment of the present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the plant growth stage detection method provided in any embodiment of the present invention.

[0171] In the process of implementing the computer program product, computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages and also conventional procedural programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0172] It should be understood that various forms of the processes shown above may be used, steps may be reordered, added, or deleted. For example, the steps recited in the present invention may be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0173] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for detecting plant growth stages, characterized in that, The method includes: Determining video features corresponding to the video to be detected and probability information of the video to be detected; wherein, the video to be detected includes plant growth information, and the probability information is used to indicate the mapping relationship between the boundary probability and time and the mapping relationship between the action probability and time in the video to be detected; Determining a candidate time interval in the video to be detected according to the probability information; wherein, the confidence level of the candidate time interval is greater than a preset confidence level threshold, and the confidence level of the candidate time interval is a value determined according to the corresponding boundary probability and action probability; Inputting the video features, the probability information, and the candidate time interval into a target neural network, and obtaining a detection result output by the target neural network; wherein, the target neural network is used to determine the detection result by combining prior information on plant growth, and the detection result includes: at least one target time interval and the plant growth stage corresponding to each target time interval.

2. The method according to claim 1, wherein The determining the video features corresponding to the video to be detected and the probability information of the video to be detected includes: Inputting each image frame of the video to be detected and the stacked optical flow field of the image frame into a first neural network, and obtaining video features output by the first neural network; wherein, the video features include features corresponding to each image frame, and the first neural network is used to respectively encode the image frame and the stacked optical flow field of the image frame and then splice them to obtain features corresponding to each image frame; Inputting the video features into a second neural network, and obtaining probability information output by the second neural network; wherein, the second neural network is used to determine the probability information according to the temporal convolutional module included therein.

3. The method according to claim 1, characterized in that, The boundary probability includes a start probability and an end probability, and the probability information includes: a start probability sequence, an end probability sequence, and an action probability sequence; The determining the candidate time interval in the video to be detected according to the probability information includes: Determining a preliminary screening start time point as a time point at which the corresponding start probability in the start probability sequence is higher than a first probability threshold and is a local extremely high probability; Determining a preliminary screening end time point as a time point at which the corresponding end probability in the end probability sequence is higher than a second probability threshold and is a local extremely high probability; Determining each preliminary screening time interval according to the preliminary screening start time point and the preliminary screening end time point; wherein, each preliminary screening time interval includes a preliminary screening start time point and a preliminary screening end time point, and the preliminary screening end time point is greater than the preliminary screening start time point; For each preliminary screening time interval, determining the confidence level of the preliminary screening time interval according to the duration of the preliminary screening time interval, the start probability, end probability, and action probability corresponding to the time points within the preliminary screening time interval; Determining the preliminary screening time interval with a corresponding confidence level greater than the preset confidence level threshold as the candidate time interval.

4. The method according to claim 3, characterized in that The determining the preliminary screening time interval with a corresponding confidence level greater than the preset confidence level threshold as the candidate time interval includes: If the initial screening time interval corresponding to a confidence level greater than the preset confidence level threshold does not contain a low confidence level time interval, then determine this initial screening time interval as the candidate time interval; wherein, the low confidence level time interval is an initial screening time interval corresponding to a confidence level less than the preset confidence level threshold. If the initial screening time interval corresponding to a confidence level greater than the preset confidence level threshold contains at least a part of a low confidence level time interval, and the overlap degree between the low confidence level time interval and this initial screening time interval is greater than the preset overlap degree threshold, then remove the time interval overlapping with the low confidence level time interval from this initial screening time interval, and determine the remaining time interval of this initial screening time interval as the candidate time interval. If the initial screening time interval corresponding to a confidence level greater than the preset confidence level threshold contains at least a part of a low confidence level time interval, and the overlap degree between the low confidence level time interval and this initial screening time interval is less than or equal to the preset overlap degree threshold, then determine this initial screening time interval as the candidate time interval.

5. The method according to any one of claims 1 to 4, characterized in that, The target neural network includes a boundary attention module, an encoder module, a decoder module, and a morphological prior constraint module connected in sequence. The step of inputting the video feature, the probability information, and the candidate time interval into the target neural network to obtain the detection result output by the target neural network includes: Input the video feature and the probability information into the boundary attention module, and input the candidate time interval into the decoder module to obtain the detection result output by the morphological prior constraint module; wherein, the morphological prior constraint module is used to verify the result output by the decoder module according to the plant growth prior information to obtain the detection result.

6. The method according to claim 5, characterized in that, The result output by the decoder module includes: each secondary screening time interval, the plant growth stage corresponding to each secondary screening time interval, and the confidence level of each secondary screening time interval, and at least one secondary screening time interval corresponds to at least two plant growth stages. The step of obtaining the detection result output by the morphological prior constraint module includes: Through the morphological prior constraint module, select a plant growth stage for the secondary screening time interval corresponding to at least two plant growth stages, sort each secondary screening time interval in chronological order to obtain an initial time interval sequence. Through the morphological prior constraint module, traverse each pair of adjacent secondary screening time intervals in the initial time interval sequence, and remove the secondary screening time intervals that do not conform to the plant growth prior information to obtain a screened time interval sequence. Through the morphological prior constraint module, according to the transition probability between each pair of adjacent secondary screening time intervals in the screened time interval sequence in the plant growth prior information, and the confidence level of the adjacent secondary screening time intervals, determine the rationality score of the screened time interval sequence. If there are unselected plant growth stages in the secondary screening time intervals corresponding to at least two plant growth stages, then through the morphological prior constraint module, return to execute the step of "selecting a plant growth stage for the secondary screening time intervals corresponding to at least two plant growth stages through the morphological prior constraint module, sorting each secondary screening time interval in chronological order to obtain an initial time interval sequence"; If there are no unselected plant growth stages in the secondary screening time intervals corresponding to at least two plant growth stages, stop the iteration; Through the morphological prior constraint module, determine the secondary screening time intervals included in the screened time interval sequence with the highest corresponding rationality score as the target time intervals, and determine the plant growth stages corresponding to the secondary screening time intervals included in the screened time interval sequence as the plant growth stages corresponding to the target time intervals.

7. The method according to claim 6, wherein The detection result also includes the confidence level corresponding to each target time interval; The method further includes: Through the morphological prior constraint module, sort each target time interval in chronological order to obtain a target time interval sequence; If there is an overlap between two adjacent target time intervals in the target time interval sequence, then through the morphological prior constraint module, traverse all splitting schemes to obtain at least two updated time interval sequences; wherein, the updated time interval sequence includes: non-overlapping target time intervals and splitting time intervals, and the splitting scheme is used to indicate the start time point and end time point of the splitting time interval, and to indicate the plant growth stage and confidence level corresponding to the splitting time interval; For each updated time interval sequence, through the morphological prior constraint module, traverse all completion schemes to obtain at least two secondary updated time interval sequences; wherein, the secondary updated time interval sequence includes: non-overlapping target time intervals, splitting time intervals and completion time intervals with no time gap between adjacent time intervals, and the completion scheme is used to indicate the plant growth stage and confidence level corresponding to the completion time interval; For each secondary updated time interval sequence, through the morphological prior constraint module, according to the transition probability of each pair of adjacent time intervals in the secondary updated time interval sequence in the plant growth prior information, and the confidence level of the adjacent time intervals, determine the rationality score of the secondary updated time interval sequence; Through the morphological prior constraint module, determine the time intervals included in the secondary updated time interval sequence with the highest corresponding rationality score as the final time intervals, and determine the plant growth stages corresponding to the time intervals included in the secondary updated time interval sequence as the plant growth stages corresponding to the final time intervals.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the plant growth stage detection method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program is used to implement the plant growth stage detection method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the computer program implements the plant growth stage detection method according to any one of claims 1 to 7.

Images