Historical information-based variable gravity particle experiment process real-time detection method

Abstract

The invention provides a variable gravity particle experiment process real-time detection method based on historical information, and relates to the technical field of video data processing. According to the method provided by the invention, a video feature vector corresponding to a target video clip is determined, and then a memory feature vector is determined according to historical feature information of a plurality of video clips before the target video clip in a plurality of continuous video clips; according to the video feature vector and the memory feature vector corresponding to the target video clip, determining a process category corresponding to the target video clip, and starting time and ending time corresponding to each process category; according to the method, in the experimental process of the spatial variable-gravity granular material, accurate detection and identification of different stages and processes of the spatial variable-gravity granular material experiment can be completed in real time, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video data processing, and in particular to a real-time detection method for a variable gravity particle experiment process based on historical information. Background Art

[0002] Variable gravity particle experiments are a key means to study the behavior characteristics of granular materials under different gravity environments. They have important scientific value and engineering significance for in-depth understanding of the basic physical mechanisms of granular media, optimizing material design, and promoting aerospace engineering applications. Under the Earth's gravity conditions, the flow, stacking, compression and other characteristics of granular materials are dominated by gravity, while in microgravity or variable gravity environments, their internal forces, interactions between particles and overall evolution laws may change significantly. This is not only related to propellant management in spacecraft propulsion systems and material control in space manufacturing, but also involves core technical issues such as in-situ resource utilization (ISRU) in deep space exploration missions such as the moon and Mars. Therefore, studying the evolution of granular materials in variable gravity environments has a far-reaching impact on promoting basic physics research, improving the technical reserves for future deep space exploration missions, and even promoting related industrial applications.

[0003] Since variable gravity particle experiments usually involve complex experimental processes, long experimental durations, and multi-stage evolution processes, how to efficiently and accurately identify experimental stages and extract key experimental information has become a core challenge for experimental data analysis and experimental control. Current experimental data processing methods often rely on offline analysis, which makes it difficult to obtain key state changes during the experiment in a timely manner, resulting in delayed experimental adjustments and affecting experimental efficiency and data quality.

[0004] Therefore, there is an urgent need for a real-time detection method for the variable gravity particle experiment process based on historical information, which can make full use of historical data during the experiment, accurately identify the status of each stage of the experiment, improve data processing efficiency, realize dynamic optimization and regulation of experimental conditions, and ensure the stability and scientific value of the experiment. The realization of this method will not only greatly improve the data utilization efficiency of variable gravity particle experiments, but also provide more intelligent and automated experimental monitoring and control means for future space science experiments. Summary of the invention

[0005] The embodiment of the present invention provides a real-time detection method for a variable gravity particle experiment process based on historical information, which can accurately detect and identify different stages of a space variable gravity particle material experiment in real time during the space variable gravity particle material experiment, thereby improving data processing efficiency.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions: In a first aspect, a method for real-time detection of a variable gravity particle experiment process based on historical information is provided, the method comprising: obtaining a target video segment of the variable gravity particle experiment, the target video segment comprising a plurality of continuous frame images, the target video segment being one of a plurality of continuous video segments of the variable gravity particle experiment; determining a first feature vector corresponding to the target video segment; performing a position encoding operation on the first feature vector and first historical feature information of k video segments arranged in sequence and located before the target video segment in a plurality of video segments included in a historical feature queue, to obtain a second feature vector corresponding to the first feature vector, and second historical feature information corresponding to the first historical feature information of each of the k video segments; and obtaining a first feature vector corresponding to the first feature vector according to the second historical feature information of each of the k video segments. The memory feature vector is determined based on the feature information of the target video clip, and the memory feature vector includes short-term historical feature information and long-term historical feature information; the short-term historical feature information is determined based on the second historical feature information of n video clips adjacent to the target video clip in the historical feature queue, and the long-term historical feature information is determined based on the second historical feature information of m video clips other than the n video clips adjacent to the target video clip in the historical feature queue, k, n and m are positive integers, and m+n=k; the experimental process detection result of the target video clip is determined by the real-time process detection unit that has been trained based on the second feature vector of the target video clip and the memory feature vector, and the experimental process detection result includes at least one process category of a variable gravity particle experiment, and the start time and end time corresponding to each process category.

[0007] In a possible implementation manner of the first aspect, the method further includes: based on a multi-layer perceptron, performing a projection operation on the experimental process detection result of the target video segment to obtain a projection feature vector corresponding to the target video segment; determining first historical feature information corresponding to the target video segment according to the projection feature vector and the first feature vector corresponding to the target video segment; and updating the historical feature queue according to the first historical feature information corresponding to the target video segment; Among them, the projection feature vector corresponding to the target video clip The formula for determining is: ; ; is the experimental process detection result of the target video clip, is the first feature vector corresponding to the target video segment, The first historical feature information corresponding to the target video segment.

[0008] In a possible implementation of the first aspect, a historical feature queue is updated according to the first historical feature information corresponding to the target video clip, including: when the target video clip is the tth of multiple consecutive video clips of a variable gravity particle experiment, and t is less than or equal to k, the first historical feature information corresponding to the target video clip is added to the historical feature queue; when the target video clip is the tth of multiple consecutive video clips of a variable gravity particle experiment, and t is greater than k, the first historical feature information corresponding to the video clip ranked first in the historical feature queue is deleted, and the first historical feature information corresponding to the target video clip is added to the historical feature queue.

[0009] In a possible implementation of the first aspect, a memory feature vector is determined based on the second historical feature information of each video clip in k video clips, including: splicing the second historical feature information of n video clips adjacent to the target video clip to obtain short-term historical feature information; dividing the second historical feature information of m video clips in the historical feature queue except the n video clips adjacent to the target video clip into x information groups, each information group includes the second historical feature information of y video clips, wherein x and y are positive integers, and x*y=m; based on a compression aggregation network, compressing and aggregating the second historical feature information of the y video clips included in each information group to obtain a compressed aggregation feature corresponding to each information group, wherein the compression aggregation network is a neural network structure based on Taylor expansion and Kolmogorov-Arnold representation theorem; splicing the compressed aggregation features corresponding to each information group to obtain long-term historical feature information; splicing the short-term historical feature information and the long-term historical feature information to obtain a memory feature vector.

[0010] In a possible implementation of the first aspect, the compression aggregation network is used to: perform feature transformation on the second historical feature information of each information group including y video clips based on Taylor polynomials to obtain transformation features corresponding to each second historical feature information; and perform aggregation processing on the transformation features corresponding to the second historical feature information included in each information group to obtain compressed aggregation features corresponding to each information group.

[0011] In a possible implementation manner of the first aspect, the trained real-time process detection unit includes a plurality of transformer decoder units connected in series, and the plurality of transformer decoder units connected in series are used to output a processing result according to an input query vector, a key vector, and a value vector; Experimental process detection results of target video clips The formula for determining is: ; ; ; ; in, is the activation function; The processing result output by the Transformer Decoder unit; is the weight for projecting the processing results output by the Transformer Decoder unit into the category space, and c is the number of process categories; is the query vector, K is the key vector, and V is the value vector; is the second feature vector of the target video clip; is the memory feature vector of the target video segment.

[0012] In a possible implementation of the first aspect, before determining the experimental process detection result of the target video segment through a trained real-time process detection unit based on the first feature vector and the memory feature vector of the target video segment, the above method also includes: obtaining a training sample set of a variable gravity particle experiment, the training sample set including multiple training samples, each training sample including a training video segment and an experimental process detection result corresponding to the training video segment; constructing a target loss function; based on the target loss function, iteratively training the real-time process detection unit to obtain a trained real-time process detection unit.

[0013] In a possible implementation of the first aspect, the target loss function for: ; ; ; in, is the balancing factor for class i in each batch, is the predicted probability that the number of video frames corresponding to time t is process category i; yes Moments belong to the process category The true label of is the balance factor; is the total number of process instances in each batch, is the mean statistic based on logarithmic transformation, is the process category in each batch The number of process instances.

[0014] The beneficial effects of the present invention are as follows: the method provided by the present invention determines the video feature vector corresponding to the target video clip, and then determines the memory feature vector according to the historical feature information of multiple video clips located before the target video clip in multiple consecutive video clips, and then determines the process category corresponding to the target video clip according to the video feature vector and the memory feature vector corresponding to the target video clip, which can accurately detect and identify the different stage processes of the spatial variable gravity granular material experiment in real time during the spatial variable gravity granular material experiment, thereby improving data processing efficiency. On the other hand, since the memory feature vector includes short-term historical feature information and long-term historical feature information, it can accurately detect the process with a shorter duration and the process with a longer duration during the spatial variable gravity granular material experiment. In addition, the method provided by the embodiment of the present invention determines the long-term historical feature information based on the neural network structure of Taylor expansion and Kolmogorov-Arnold representation theorem, which can effectively reduce the amount of calculation while retaining the historical feature information, thereby saving computing resources and reducing computing costs. Finally, the method provided by the embodiment of the present invention designs a target loss function. For the process category with a shorter duration during the experiment, the target loss function can dynamically adjust the balance factor strategy, so that the real-time process detection unit can focus on the process category with a shorter duration during the training process, thereby effectively improving the accuracy of process detection.

[0015] In a second aspect, the present invention provides a real-time detection system for a variable gravity particle experiment process based on historical information, the system comprising: a video acquisition unit, for acquiring a target video segment of the variable gravity particle experiment, the target video segment comprising a continuous multi-frame image, the target video segment being one of a plurality of continuous video segments of the variable gravity particle experiment; a feature extraction unit, for determining a first feature vector corresponding to the target video segment; a position encoding unit, for performing a position encoding operation on the first feature vector and first historical feature information of k video segments arranged in sequence and located before the target video segment in a plurality of video segments included in a historical feature queue, to obtain a second feature vector corresponding to the first feature vector, and second historical feature information corresponding to the first historical feature information of each of the k video segments; a vector determination unit, for determining a first feature vector corresponding to the target video segment according to the k video segments; The memory feature vector is determined based on the second historical feature information of each video clip in the video clip, and the memory feature vector includes short-term historical feature information and long-term historical feature information; the short-term historical feature information is determined based on the second historical feature information of n video clips adjacent to the target video clip in the historical feature queue, and the long-term historical feature information is determined based on the second historical feature information of m video clips other than the n video clips adjacent to the target video clip in the historical feature queue, k, n and m are positive integers, and m+n=k; a process detection unit is used to determine the experimental process detection result of the target video clip through the real-time process detection unit that has been trained according to the second feature vector of the target video clip and the memory feature vector, and the experimental process detection result includes at least one process category of a variable gravity particle experiment, and the start time and end time corresponding to each process category.

[0016] According to a third aspect, an electronic device is provided, comprising a memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the processor, the electronic device executes a method as in any implementation of the first aspect.

[0017] According to a fourth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method in any implementation of the first aspect.

[0018] According to a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method in any implementation of the first aspect.

[0019] It can be understood that the beneficial effects that can be achieved by the system of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A process schematic diagram of a method for real-time detection of a variable gravity particle experiment process based on historical information provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention; Figure 3 A flow chart of a method for real-time detection of a variable gravity particle experiment process based on historical information provided by an embodiment of the present invention; Figure 4 A flow chart of another method for real-time detection of variable gravity particle experiment process based on historical information provided by an embodiment of the present invention; Figure 5 A flow chart of another method for real-time detection of variable gravity particle experiment process based on historical information provided by an embodiment of the present invention; Figure 6 A flow chart of another method for real-time detection of variable gravity particle experiment process based on historical information provided by an embodiment of the present invention; Figure 7 A schematic diagram of the process categories and duration of a spatial variable gravity granular material experiment for a validation data set provided by an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a detection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution in the embodiment of the present invention will be described below in conjunction with the accompanying drawings in the embodiment of the present invention. In the description of the present invention, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B; the "or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in the description of the present invention, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items.

[0022] In addition, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0023] Meanwhile, in the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being better or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0024] Variable gravity particle experiments are a key means to study the behavior characteristics of granular materials under different gravity environments. They have important scientific value and engineering significance for in-depth understanding of the basic physical mechanisms of granular media, optimizing material design, and promoting aerospace engineering applications. Under the Earth's gravity conditions, the flow, stacking, compression and other characteristics of granular materials are dominated by gravity, while in microgravity or variable gravity environments, their internal forces, interactions between particles and overall evolution laws may change significantly. This is not only related to propellant management in spacecraft propulsion systems and material control in space manufacturing, but also involves core technical issues such as in-situ resource utilization (ISRU) in deep space exploration missions such as the moon and Mars. Therefore, studying the evolution of granular materials in variable gravity environments has a far-reaching impact on promoting basic physics research, improving the technical reserves for future deep space exploration missions, and even promoting related industrial applications.

[0025] Since variable gravity particle experiments usually involve complex experimental processes, long experimental durations, and multi-stage evolution processes, how to efficiently and accurately identify experimental stages and extract key experimental information has become a core challenge for experimental data analysis and experimental control. Current experimental data processing methods often rely on offline analysis, which makes it difficult to obtain key state changes during the experiment in a timely manner, resulting in delayed experimental adjustments and affecting experimental efficiency and data quality.

[0026] Therefore, there is an urgent need for a real-time detection method for the variable gravity particle experiment process based on historical information, which can make full use of historical data during the experiment, accurately identify the status of each stage of the experiment, improve data processing efficiency, realize dynamic optimization and regulation of experimental conditions, and ensure the stability and scientific value of the experiment. The realization of this method will not only greatly improve the data utilization efficiency of variable gravity particle experiments, but also provide more intelligent and automated experimental monitoring and control means for future space science experiments.

[0027] In view of this, an embodiment of the present invention provides a real-time detection method for a variable gravity particle experiment process based on historical information, the method comprising: obtaining a target video segment of the variable gravity particle experiment, the target video segment comprising a continuous multi-frame image, the target video segment being one of a plurality of continuous video segments of the variable gravity particle experiment; determining a first feature vector corresponding to the target video segment; performing a position encoding operation on the first feature vector and first historical feature information of k video segments arranged in sequence and located before the target video segment in a plurality of video segments included in a historical feature queue, to obtain a second feature vector corresponding to the first feature vector, and second historical feature information corresponding to the first historical feature information of each of the k video segments; and obtaining a second feature vector corresponding to the first historical feature information of each of the k video segments according to the second historical feature information of each of the k video segments. The historical feature information determines a memory feature vector, and the memory feature vector includes short-term historical feature information and long-term historical feature information; the short-term historical feature information is determined based on the second historical feature information of n video clips adjacent to the target video clip in the historical feature queue, and the long-term historical feature information is determined based on the second historical feature information of m video clips in the historical feature queue except the n video clips adjacent to the target video clip, k, n and m are positive integers, and m+n=k; the experimental process detection result of the target video clip is determined by a real-time process detection unit that has been trained according to the second feature vector of the target video clip and the memory feature vector, and the experimental process detection result includes at least one process category of a variable gravity particle experiment, and the start time and end time corresponding to each process category.

[0028] The method provided by the present invention determines the video feature vector corresponding to the target video clip, and then determines the memory feature vector according to the historical feature information of multiple video clips located before the target video clip in multiple consecutive video clips, and then determines the process category corresponding to the target video clip according to the video feature vector and the memory feature vector corresponding to the target video clip. During the space variable gravity granular material experiment, accurate detection and identification of different stage processes of the space variable gravity granular material experiment can be completed in real time, thereby improving data processing efficiency.

[0029] In some embodiments, a real-time detection method for a variable gravity particle experiment process based on historical information provided by an embodiment of the present invention may be executed by a real-time detection system 100 for a variable gravity particle experiment process based on historical information (hereinafter referred to as detection system 100 ).

[0030] For example, see Figure 1 , Figure 1A process schematic diagram of a real-time detection method for a variable gravity particle experimental process based on historical information provided by an embodiment of the present invention, first, a target video segment (the t-th one of the multiple continuous video segments sorted based on a time series) is obtained from multiple continuous video segments, and then a first feature vector corresponding to the target video segment is determined by a feature extractor, and then the detection system 100 performs position encoding according to the first historical feature information and the first feature vector of k video segments (t-1th to tkth video segments) included in the historical feature queue to obtain the second historical feature information corresponding to each first historical feature information and the second feature vector corresponding to each first feature vector, and then the detection system constructs short-term historical feature information according to the second historical feature information of n video segments adjacent to the target video segment, and constructs long-term historical feature information according to the second historical feature information of m video segments in the historical feature queue except the n video segments adjacent to the target video segment, and then constructs a memory feature vector through the short-term historical feature information and the long-term historical feature information, and finally determines the experimental process detection result of the target video segment through the real-time process detection unit according to the memory feature vector and the second feature vector. In addition, the method provided by the embodiment of the present invention generates first historical feature information of the target video clip according to the experimental process detection results of the target video clip and the first feature vector, and updates the historical feature queue according to the first historical feature information of the target video clip (equivalent to adding the first historical feature information of the tth video clip and removing the first historical feature information of the tkth video clip). The updated historical feature queue includes the first historical feature information of k video clips (the tth to the t-k+1th video clips).

[0031] As an example, the detection system 100 can be any electronic device 200 with data processing capabilities, such as a general-purpose computer, a personal computer, a laptop computer, a switch or a tablet computer, etc. The specific implementation method of the detection system 100 is not limited here.

[0032] Figure 2 The hardware structure diagram of the electronic device provided by the embodiment of the present invention is shown. The electronic device 200 includes a processor 210, a memory 220 and a communication interface 230.

[0033] The processor 210 may include one or more processing cores. The processor 210 uses various interfaces and lines to connect various parts in the electronic device 200, and executes various functions and processes data of the electronic device 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 220, and calling data stored in the memory 220. Optionally, the processor 210 can be implemented in at least one hardware form of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA).

[0034] The memory 220 may include a random access memory (RAl) or a read-only memory (ROL). Optionally, the memory 220 includes a non-transitory computer-readable storage medium (non-transitory colputer-readable storage lediul). The memory 220 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 220 may include a program storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a video acquisition function, a feature extraction function, and a process detection function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.

[0035] The communication interface 230 is used to communicate with other devices, equipment or communication networks, such as data storage devices, image processing equipment or Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0036] In physical implementation, the above-mentioned components (such as processor 210, memory 220 and communication interface 230) can be components in the same device (such as a laptop computer). Alternatively, at least two of the components can be set in the same device, that is, as different components in a device, such as a deployment method similar to devices or components in a distributed system.

[0037] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments of the present invention, the electronic device 200 may include more or fewer components than those illustrated, or combine certain components, or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0038] A real-time detection method for a variable gravity particle experiment process based on historical information provided by an embodiment of the present invention is described below in conjunction with the accompanying drawings.

[0039] Figure 3 A flowchart of a method for real-time detection of a variable gravity particle experiment process based on historical information provided by an embodiment of the present invention. Optionally, the method can be Figure 1 The detection system 100 shown, that is, Figure 2 The electronic device 200 shown is executed. The method may include the following steps: S1. Obtain the target video clip of the variable gravity particle experiment.

[0040] Specifically, the target video segment includes a plurality of continuous frames of images, and the target video segment is one of a plurality of continuous video segments of the variable gravity particle experiment.

[0041] The variable gravity particle experiment includes multiple process stages, each of which corresponds to a different process category. For example, the process categories of the process stages of the variable gravity particle experiment include: baffle left movement before vibration, baffle right movement before vibration, baffle vibration, baffle stop vibration, baffle reset after vibration, baffle left movement before collapse, variable gravity before collapse, baffle front pressure before collapse, and collapse. It should be noted that for the variable gravity particle experiment, multiple process stages do not necessarily occur continuously. There may be a static process stage (also called background) between two adjacent process stages. In the present invention, the process category corresponding to the static process stage is no category.

[0042] It should be understood that the above variable gravity particle experiment is only an exemplary description, and the method provided in the embodiment of the present invention can also be used to perform process detection on experiments of other types and fields, and the embodiment of the present invention does not impose any particular limitation on this.

[0043] Specifically, the multiple continuous video clips are video clips arranged in chronological order, and each frame of image included in each video clip has a corresponding time stamp, and the time stamp is used to represent the acquisition time of each image.

[0044] In one example, experimental video data of a variable gravity particle experiment includes 20,000 frames of images, which are divided into 4,000 video segments with 5 frames of images as one video segment, and the target video segment is the 100th video segment of the 4,000 video segments arranged continuously in chronological order.

[0045] It should be noted that the number of video clips included in the above variable gravity particle experiment and the number of image frames included in each video clip are only exemplary. The variable gravity particle experiment can also include a greater or lesser number of video clips, and each video clip can include a greater or lesser number of image frames. The embodiment of the present invention does not impose any particular limitation on this.

[0046] S2. Determine a first feature vector corresponding to the target video segment.

[0047] In a possible implementation, the detection system 100 initializes the weights of a vision transformer (ViT) according to the pre-trained model weights, and then uses ViT as a feature encoder to extract a first feature vector corresponding to the target video segment.

[0048] Specifically, ViT is a neural network model that applies the Transformer architecture to computer vision tasks. ViT divides each frame of the target video clip into multiple image blocks, and uses the multiple image blocks as sequence data. It extracts image features through the Transformer architecture to obtain the first feature vector corresponding to the target video clip.

[0049] The method provided by the embodiment of the present invention uses ViT to determine the first feature vector corresponding to the target video clip. Compared with the feature extraction method of the feature extraction network of the CNN architecture in the related art, the method provided by the embodiment of the present invention can effectively improve the accuracy and stability of feature extraction.

[0050] S3. Perform a position encoding operation on the first feature vector and the first historical feature information of k video clips arranged in sequence and located before the target video clip in multiple video clips included in the historical feature queue to obtain a second feature vector corresponding to the first feature vector, and second historical feature information corresponding to the first historical feature information of each video clip in the k video clips.

[0051] Specifically, the historical feature queue includes the first historical feature information of k video segments that are arranged in sequence and are located before the target video segment in multiple video segments. That is, the length of the historical feature queue is k. In other words, when the target video segment is the tth video segment among multiple video segments that are arranged continuously in chronological order, and t is greater than k, the historical feature queue includes the first historical feature information corresponding to the tkth video segment to the t-1th video segment that are arranged in sequence. Combined with the above example, when k=80, the target video segment is the 100th video segment among 200 video segments that are arranged continuously in chronological order, then the historical feature queue includes the first historical feature information corresponding to the 20th video segment to the 99th video segment that are arranged in sequence.

[0052] Among them, since the first feature vector corresponding to the target video clip does not include temporal position information, it is necessary to perform position encoding operation on the first feature vector and the first historical feature information of k video clips arranged in sequence and located before the target video clip in multiple video clips included in the historical feature queue, so as to obtain the second feature vector corresponding to the first feature vector, and the second historical feature information corresponding to the first historical feature information of each video clip in the k video clips.

[0053] The calculation formula is: ; It is stored in the historical feature queue HFQ The first historical feature information corresponding to the video clip, is the first eigenvector corresponding to the target video segment, After position encoding The second historical feature information corresponding to the video clip, is the second feature vector corresponding to the target video segment after position encoding, It refers to the position encoding operation.

[0054] In one example, the position encoding operation is Sinusoidal Position Encoding.

[0055] S4. Determine a memory feature vector according to the second historical feature information of each video segment in the k video segments, where the memory feature vector includes short-term historical feature information and long-term historical feature information.

[0056] Specifically, the short-term historical feature information is determined based on the second historical feature information of n video clips adjacent to the target video clip in the historical feature queue, and the long-term historical feature information is determined based on the second historical feature information of m video clips in the historical feature queue except the n video clips adjacent to the target video clip, k, n and m are positive integers, and m+n=k.

[0057] It should be noted that after the position encoding The second historical feature information corresponding to the video clip As a feature memory vector, When the value of is large, the amount of calculation during model decoding will increase, reducing the decoding efficiency. In addition, simply increasing the length of historical feature information cannot effectively improve the real-time detection accuracy of the model for the experimental process. To this end, the method provided in the embodiment of the present invention converts the position-encoded The second historical feature information corresponding to the video clip Divided into long-term historical feature information and short-term historical feature information.

[0058] In some embodiments, see Figure 4 The above S4 specifically includes the following steps: S41, concatenating the second historical feature information of n video segments adjacent to the target video segment to obtain short-term historical feature information; Specific, short-term historical feature information The formula for determining is: ; It refers to the second historical feature information of n video segments adjacent to the target video segment in the historical feature queue HFQ.

[0059] The method provided by the embodiment of the present invention obtains short-term historical feature information by splicing the second historical feature information of n video clips adjacent to the target video clip, which can provide more accurate short-term historical information dependency for the prediction of the process category of the target video clip.

[0060] S42: Divide the second historical feature information of the m video segments in the historical feature queue except the n video segments adjacent to the target video segment into x information groups, each information group including the second historical feature information of y video segments.

[0061] Where x and y are positive integers, and x*y=m.

[0062] S43. Based on the compression aggregation network, compress and aggregate the second historical feature information of the y video clips included in each information group to obtain the compression aggregation feature corresponding to each information group.

[0063] Among them, the compressed aggregation network is a neural network structure based on Taylor expansion and Kolmogorov-Arnold representation theorem.

[0064] Specifically, for long-term historical feature information, in order to compress the feature length and highlight more important long-term historical feature information, the method provided by the embodiment of the present invention adopts The compression aggregation network of the layer compresses and aggregates the long-term historical feature information. Each information group includes the second historical feature information of y video clips and uses the same compression aggregation network for compression aggregation, that is: ; ; in, Each information group includes the second historical feature information of y video clips, It refers to the compressed aggregation feature corresponding to the xth information group. TaylorKAN represents the compressed aggregation operation.

[0065] In one example, the number of layers of the compression aggregation network is set to 3.

[0066] In some embodiments, the compression aggregation network is used to: Based on Taylor polynomials, feature transformation is performed on the second historical feature information of y video clips in each information group to obtain the transformation features corresponding to each second historical feature information; and aggregation processing is performed on the transformation features corresponding to the second historical feature information included in each information group to obtain the compressed aggregation features corresponding to each information group.

[0067] Specifically, the compressed aggregation network can also be called TaylorKAN network, which is a variant of Kolmogorov-Arnold (KAN) network structure. The idea of ​​KAN network structure is based on Kolmogorov-Arnold representation theorem, which states that if is any multivariable continuous function defined on a bounded domain, then the function It can be expressed as a two-layer nested addition of a finite number of single-variable, continuous functions, that is, any multivariable continuous function can be expressed as a combination of single-variable continuous functions and addition operations.

[0068] Compared with the MLP network in related technologies, the MLP places a fixed activation function on the nodes (neurons), while the KAN network places a learnable activation function on the edges (weights). This design allows each weight parameter in the KAN network to be replaced by a single variable function. These functions are usually parameterized in the form of spline functions, which provides extremely high flexibility and can simulate complex functions with fewer parameters, enhancing the interpretability of KAN.

[0069] KAN networks usually use B-spline functions as activation functions to adaptively model complex relationships in data, and the activation function usually includes B-spline basis functions.

[0070] The compression aggregation network provided by the embodiment of the present invention is a variant of the KAN network that uses Taylor polynomials instead of B-spline basis functions as activation functions, and uses Taylor polynomials of different orders to perform local approximation and other operations on input data to generate local high-order nonlinear features.

[0071] Among them, Taylor polynomial The formula is: ; is the expansion level.

[0072] S44, concatenating the compressed aggregate features corresponding to each information group to obtain long-term historical feature information; Specific, long-term historical feature information The formula for determining is: ; S45. Concatenate the short-term historical feature information and the long-term historical feature information to obtain a memory feature vector.

[0073] Specifically, remember the feature vector The formula for determining is: ; ; ; is the memory feature vector The length of the historical information input into the real-time process detection unit after training is completed. It is the long-term historical feature information after compression and aggregation Length, It is short-term historical characteristic information Length, is the length of the historical feature queue HFQ. x is the number of information groups.

[0074] In one example, Set to 50, is 10, is 8, is 5, It can also be understood that: the historical feature queue includes historical feature information corresponding to 50 video clips, and the short-term historical feature information is obtained by performing a splicing operation based on the second historical feature information corresponding to 10 video clips adjacent to the target video clip. Then the second historical feature information corresponding to the remaining 40 video clips is grouped into 8 information groups, each of which includes the second historical feature information corresponding to 5 video clips. Then the 8 information groups are compressed and aggregated respectively, and finally spliced ​​to obtain the long-term historical feature information. Then the short-term historical feature information and long-term historical characteristic information Perform concatenation operation to obtain the memory feature vector .

[0075] S5. Determine the experimental process detection result of the target video segment through the trained real-time process detection unit according to the second feature vector and the memory feature vector of the target video segment.

[0076] The experimental process detection result includes at least one process category of a variable gravity particle experiment, and a start time and an end time corresponding to each process category.

[0077] Specifically, the trained real-time process detection unit determines the experimental process detection results of the target video clip, which can also be understood as determining the process category corresponding to each frame of the target video clip, as well as the start time and end time corresponding to each process category. Among them, the process categories include left movement of the baffle before vibration, right movement of the baffle before vibration, baffle vibration, baffle stop vibration, baffle reset after vibration, left movement of the baffle before collapse, gravity change before collapse, front pressure of the baffle before collapse, and collapse. The process category also includes no category (stationary process stage).

[0078] Exemplarily, the target video clip includes multiple consecutive frames of images taken when the variable gravity particle experiment A is in the static process stage. Then the experimental process detection result of the target video clip includes a process category of no category, the start time corresponding to no category is the timestamp of the first frame of the target video clip, and the end time corresponding to no category is the timestamp of the last frame of the target video clip.

[0079] In another example, the target video clip includes multiple consecutive frames of images taken when the variable gravity particle experiment A is in the static process stage and the baffle is vibrating. Then the experimental process detection results of the target video clip include process categories of no category and baffle vibration. The start time corresponding to no category is the timestamp of the first frame of the target video clip, and the timestamp corresponding to the frame image when the baffle starts to vibrate is the start time corresponding to the baffle vibration; the end time corresponding to the baffle vibration is the timestamp of the last frame of the target video clip.

[0080] Furthermore, the trained real-time process detection unit also obtains the experimental process detection result of the variable gravity particle experiment according to the experimental process detection result of each video segment in the continuous multiple video segments of the variable gravity particle experiment. The experimental process detection result includes at least one process category of the variable gravity particle experiment, and the start time and end time corresponding to each process category.

[0081] In some embodiments, the trained real-time process detection unit includes a plurality of serially connected Transformer Decoder units, and the plurality of serially connected Transformer Decoder units are used to output processing results according to the input query vector, key vector, and value vector.

[0082] Experimental process detection results of target video clips The formula for determining is: ; ; ; ; in, is the activation function; The processing result output by the Transformer Decoder unit; is the weight for projecting the processing results output by the Transformer Decoder unit into the category space, and c is the number of process categories; is the query vector, K is the key vector, and V is the value vector; is the second feature vector of the target video clip; is the memory feature vector of the target video segment.

[0083] In one possible implementation, see Figure 5 Before the above S5, the method provided by the embodiment of the present invention further includes: S51. Obtain a training sample set of a variable gravity particle experiment, where the training sample set includes a plurality of training samples, and each training sample includes a training video segment and an experimental process detection result corresponding to the training video segment.

[0084] S52. Construct a target loss function.

[0085] Objective loss function for: ; ; ; in, is the balancing factor for class i in each batch, is the predicted probability that the number of video frames corresponding to time t is process category i; yes Moments belong to the process category The true label of is the balance factor; is the total number of process instances in each batch, is the mean statistic based on logarithmic transformation, is the process category in each batch The number of process instances.

[0086] Since the video process detection data of the variable gravity particle experiment has the problem of category imbalance, and the real-time process detection based on the expansion of motion detection will aggravate this problem, the method proposed in the embodiment of the present invention can alleviate the category imbalance problem by calculating the Focal Loss function of the balance factor based on small batch samples, thereby improving the detection accuracy.

[0087] S53. Based on the target loss function, iteratively train the real-time process detection unit to obtain a trained real-time process detection unit.

[0088] It can be seen from the above S1-S5 that the method provided by the present invention determines the video feature vector corresponding to the target video clip, and then determines the memory feature vector based on the historical feature information of multiple video clips located before the target video clip in multiple consecutive video clips, and then determines the process category corresponding to the target video clip based on the video feature vector and the memory feature vector corresponding to the target video clip. During the space variable gravity granular material experiment, accurate detection and identification of different stage processes of the space variable gravity granular material experiment can be completed in real time, thereby improving data processing efficiency.

[0089] In some embodiments, see Figure 6 , the method provided by the embodiment of the present invention further includes: S61. Based on the multi-layer perceptron, a projection operation is performed on the experimental process detection result of the target video segment to obtain a projection feature vector corresponding to the target video segment.

[0090] S62: Determine first historical feature information corresponding to the target video segment according to the projection feature vector and the first feature vector corresponding to the target video segment.

[0091] S63: Update the historical feature queue according to the first historical feature information corresponding to the target video segment.

[0092] Among them, the projection feature vector corresponding to the target video clip The formula for determining is: ; ; is the experimental process detection result of the target video clip, is the first feature vector corresponding to the target video segment, The first historical feature information corresponding to the target video segment.

[0093] In a possible implementation, the above S63 specifically includes the following steps: When the target video segment is the tth of multiple consecutive video segments of the variable gravity particle experiment, and t is less than or equal to k, the first historical feature information corresponding to the target video segment is added to the historical feature queue; when the target video segment is the tth of multiple consecutive video segments of the variable gravity particle experiment, and t is greater than k, the first historical feature information corresponding to the video segment ranked first in the historical feature queue is deleted, and the first historical feature information corresponding to the target video segment is added to the historical feature queue.

[0094] Specifically, when the target video segment is the t-th video segment and t is greater than k, the length of historical features that can be stored in the historical feature queue HFQ is Therefore, the historical feature queue HFQ stores to When the process category detection of the target video segment t is completed, the historical feature queue HFQ will The first historical feature corresponding to the video clip Delete and add the first historical feature information of the target video segment t to the historical feature queue HFQ, and the first historical feature information of the target video segment t is obtained by adding the first feature vector of the target video segment t and the projected feature vector obtained by an MLP projection layer through the detection result of the experimental process.

[0095] In addition, for the first video clip of the plurality of consecutive video clips, the detection system 100 initializes the historical feature queue HFQ, initializes the historical feature queue HFQ based on the feature vector of all zeros, and sets the length of the historical feature queue HFQ .

[0096] The beneficial effects of a real-time detection method for a variable gravity particle experiment process based on historical information provided by an embodiment of the present invention are exemplarily described below with reference to an example.

[0097] Exemplarily, the method provided by the embodiment of the present invention verifies the beneficial effects of the method provided by the embodiment of the present invention through a verification data set, which is composed of experimental videos of the variable gravity experimental cabinet granular material warehouse A collected by a panoramic camera located in the space laboratory, including 207 video clips, each of which captures rich experimental process details at a resolution of 1920×1080 and a frame rate of 25 frames per second, and the average video length reaches 9.4 minutes. In the 207 video clips, a total of 9 different types of process categories are marked, totaling 939 process instances, and the average duration of each instance is approximately 29 seconds. In addition, there is no temporal overlap between all instances.

[0098] See also Figure 7 , Figure 7 A schematic diagram of the process categories and durations of a spatial variable gravity granular material experiment for a validation data set provided by an embodiment of the present invention. The validation data set is mostly long processes, and the average duration of some processes is significantly longer. For example, the average duration of the baffle vibration process can reach 50 seconds. At the same time, some process durations also vary significantly. Compared with the baffle vibration, the average duration of the left movement of the baffle before vibration is only 1.5 seconds.

[0099] In this example, the method provided by the embodiment of the present invention is verified based on the evaluation indicators Mean Average Precision (mAP) and Average Precision (AP), and the mAP value with a step size of 0.1 on tIoU=[0.3,0.7] is compared, where tIoU refers to the temporal intersection-over-union ratio.

[0100] Average precision is an evaluation indicator for a single process category. It is used to measure the average precision at different recall levels, while mAP is the average of APs for all process categories. Specifically, when calculating AP, the prediction results are sorted according to their confidence, and then the precision at different recall thresholds is calculated. Finally, the average precision is obtained by integration or interpolation.

[0101] AP is usually calculated using the 11-point interpolation method, that is, the maximum precision values ​​at the 11 points with recall rates of 0, 0.1, 0.2, ..., 1 are averaged, and mAP is the average of the AP of all process categories, that is: Where C is the number of process categories.

[0102] tIoU is used to measure the overlap between the predicted experimental process fragment and the actual experimental process fragment. It is the ratio of the intersection area of ​​the predicted experimental process fragment and the actual experimental process fragment at time t to the union area. The higher the tIoU value, the closer the predicted result is to the actual result. The calculation of tIoU is: ; When calculating mAP, different tIoU thresholds are usually set. The prediction is considered correct only when the tIoU between the predicted result and the true result is greater than or equal to the threshold.

[0103] In addition, in order to evaluate the real-time process detection algorithm, the number of frames processed per second (FPS) is also used as an evaluation indicator. FPS refers to the number of frames processed per second, which is generally used to measure the efficiency of the model in processing images or video frames. Here, FPS is used to measure the efficiency of the process detection method provided by the present invention. The calculation formula of FPS is: ; in, is the total number of frames processed, It is the total time (in seconds) used to process these frames. FPS measures the speed at which the model processes video frames and reflects the real-time processing capability of the algorithm.

[0104] Specifically, a comparative experiment is conducted between the embodiment of the present invention and the SimOn model in the related art. The results of the comparative experiment are shown in Table 1. It should be noted that the SimOn model is a neural network structure that uses two kinds of context information as a complete context for process detection. The SimOn model does not predict the action score of each frame, but directly generates the action category probability, thereby realizing multiple action prediction.

[0105] The experimental results are shown in Table 1, in which I3D (RGB+Optical Flow) with an optical flow branch and ViT (RGB) with only RGB are used as feature extractors to determine the first feature vector. In addition, the dimension of the video features extracted by I3D is 2048, and the dimension of the video features extracted by ViT is 1408.

[0106] Table 1 As can be seen from Table 1, the method provided by the embodiment of the present invention uses ViT as the feature encoder, and has achieved significant improvement under different tIoU thresholds. The detection accuracy mAP has reached 82.79%. Compared with the SimOn model that uses ViT to extract video features, the mAP has increased by about 11.71%, indicating that the proposed LSH-RPD method can better predict the process category of the experiment based on the constructed long-term and short-term historical information, proving the effectiveness of the technical solution of the present invention.

[0107] Furthermore, in order to verify the real-time reasoning speed of the technical solution of the present invention, an FPS comparison experiment was conducted on the validation data set of the space variable gravity granular material experiment. The experimental results are shown in Table 2, where all experiments use the video features (first eigenvector) extracted by ViT. In addition, the parameter quantities and FPS in the table do not include the ViT part.

[0108] Table 2 It can be seen from Table 2 that compared with the SimOn model, although the method proposed in the embodiment of the present invention introduces a compression aggregation network to compress and aggregate long-term historical feature information, the parameter amount is only increased by about 0.13M, and the FPS does not drop too much, reaching 90.00. Therefore, on the whole, the technical solution of the present invention can complete reasoning at a faster speed while ensuring highly accurate detection results, thereby achieving a relative balance between detection accuracy and reasoning speed.

[0109] Furthermore, in order to explore the influence of the historical feature length on the detection accuracy of the experimental process, the method provided in the embodiment of the present invention also conducts an ablation experiment of the historical feature length. The experimental results are shown in Table 3, where: refers to the length of short-term historical information, refers to the length of long-term historical information, It refers to the length of the memory vector that is finally input to the real-time process detection unit. All experiments use ViT as the Encoder to extract video features. In addition, a 3-layer compression aggregation network is used to compress and aggregate long-term historical information.

[0110] Table 3 As can be seen from Table 3, when the short-term history information length is 7, the overall detection accuracy mAP of the SimOn model reaches 71.08%. However, when the short-term history information length is increased to 15, the overall detection accuracy mAP (71.09%) does not improve significantly, indicating that simply increasing the history information length cannot significantly improve the detection capability of the model.

[0111] Based on the SimOn model, a historical feature queue HFQ is constructed, and the historical feature information is divided into short-term historical information and long-term historical information. The short-term historical information is not processed, and the long-term historical information is compressed and aggregated. The length of the short-term historical information is 10, and the length of the long-term historical information is 40. It is compressed and aggregated to 5 through the TaylorKAN network to ensure that the length of the memory vector input to the real-time process detection unit is still 15. The detection accuracy of the final model at different tIoU thresholds is significantly improved, and the overall detection accuracy mAP reaches 80.66%, which is compared with the SimOn model (71.09%) with the same memory feature vector length, The mAP is improved by about 9.57%. It shows that the use of the proposed method for constructing long-term and short-term historical information can effectively improve the detection capability of the model without increasing the computational complexity of the decoder, proving the effectiveness of the technical solution of the present invention.

[0112] Furthermore, in order to verify the effectiveness of each module in the technical solution of the present invention, the method provided by the embodiment of the present invention also performs a module ablation experiment based on the verification data set. The experimental results are shown in Table 4, where all experiments use ViT as the Encoder to extract video features, Aggregation layers refers to the use of a network to compress and aggregate long-term historical information under the premise of using a historical feature queue to store long-term and short-term historical information, and loss refers to the use of the proposed balance factor based on small batch samples. Focal Loss function, in addition, the length of the memory feature vector input to the trained real-time process detection unit is 15.

[0113] It can be seen from Table 4 that whether using MLP or the more interpretable TaylorKAN network to compress and aggregate long-term historical information, the detection accuracy of the model for the experimental process can be improved. Based on the SimOn model, compared with the use of MLP, the detection accuracy mAP of the TaylorKAN network compression and aggregation method is improved by about 2.2%, indicating that compared with MLP, the more interpretable TaylorKAN network can more effectively compress and aggregate long-term historical information, obtain local approximate high-order nonlinear features, and highlight more important long-term historical features, thereby improving the process detection capability of the model.

[0114] It should be noted that although the process categories of the variable gravity particle experiment are relatively balanced (e.g. Figure 7As shown, however, the method provided by the present invention performs real-time detection of the process category of the experiment based on different video clips, and can only make predictions based on the current target video clip (current frame) during the detection process, and cannot view the future (after the target video clip in time sequence) video clips (future frames). Therefore, during the detection process, the experimental video data of the variable gravity particle experiment also has the problem of category imbalance. In order to alleviate this problem, the technical solution of the present invention uses a balance factor based on small batch samples to calculate the balance factor. Focal Loss function (target loss function) shows that after using the proposed target loss function, the detection accuracy mAP of the model reaches 82.79%. Compared with the method of using TaylorKAN to compress and aggregate long-term historical information (80.66%), mAP is improved by about 2.13%, indicating that the target loss function does effectively alleviate the problem of category imbalance in the space science experiment video process detection dataset, proving the effectiveness of the technical solution of the present invention.

[0115] Table 4 The above mainly introduces the scheme of the embodiment of the present invention from the perspective of the method. It is understandable that, in order to realize the above functions, the detection system 100 includes at least one of the hardware structure and software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiment of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present invention.

[0116] The embodiment of the present invention can divide the detection system 100 into functional units according to the above method example. For example, the detection system 100 can be divided into functional units corresponding to various functions, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present invention is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0117] For example, Figure 8A schematic diagram of the hardware structure of a detection system provided by an embodiment of the present invention is shown. The detection system 100 includes: a video acquisition unit 110, used to acquire a target video segment of a variable gravity particle experiment, the target video segment includes a continuous multi-frame image, and the target video segment is one of a plurality of continuous video segments of the variable gravity particle experiment; a feature extraction unit 120, used to determine a first feature vector corresponding to the target video segment; a position encoding unit 130, used to perform a position encoding operation on the first feature vector and the first historical feature information of k video segments arranged in sequence and located before the target video segment in a plurality of video segments included in the historical feature queue, to obtain a second feature vector corresponding to the first feature vector, and second historical feature information corresponding to the first historical feature information of each of the k video segments; a vector determination unit 140, used to determine a first feature vector corresponding to the target video segment according to the first historical feature information of each of the k video segments. The memory feature vector is determined based on the second historical feature information of n video clips adjacent to the target video clip in the historical feature queue, and the long-term historical feature information is determined based on the second historical feature information of m video clips in the historical feature queue except the n video clips adjacent to the target video clip, k, n and m are positive integers, and m+n=k; the process detection unit 150 is used to determine the experimental process detection result of the target video clip through the real-time process detection unit that has been trained according to the second feature vector of the target video clip and the memory feature vector, and the experimental process detection result includes at least one process category of a variable gravity particle experiment, and the start time and end time corresponding to each process category.

[0118] It should be understood that the specific description of the above optional methods can refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any detection system 100 provided above and the description of the beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0119] The embodiment of the present invention further provides a computer-readable storage medium, in which at least one computer instruction is stored, and the at least one computer instruction is loaded and executed by a processor to implement the methods of the above embodiments. For the explanation of the relevant contents and the description of the beneficial effects in any of the above-mentioned computer-readable storage media, reference can be made to the above-mentioned corresponding embodiments, which will not be repeated here.

[0120] The embodiment of the present invention further provides a chip. The chip integrates a control circuit and one or more ports for implementing the functions of the above detection system 100. Optionally, the functions supported by the chip can be referred to above and will not be described in detail here.

[0121] Those skilled in the art will appreciate that all or part of the steps of the above embodiments can be implemented by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a random access memory, etc. The above-mentioned processing unit or processor can be a central processing unit, a general-purpose processor, a specific circuit structure (application specific integrated circuit, ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (field prograllable gatearray, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0122] The embodiment of the present invention also provides a computer program product including instructions, when the instructions are executed on a computer, the computer executes any one of the methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD), etc.

[0123] It should be noted that the above-mentioned devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as but not limited to the above-mentioned memories, computer-readable storage media, and communication chips, etc., are all non-transitory. Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0124] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A real-time detection method for variable gravity particle experiment process based on historical information, characterized in that: The method comprises: Acquire a target video segment of a variable gravity particle experiment, wherein the target video segment includes a plurality of continuous frames of images, and the target video segment is one of a plurality of continuous video segments of the variable gravity particle experiment; Determining a first feature vector corresponding to the target video segment; Performing a position encoding operation on the first feature vector and first historical feature information of k video segments in the plurality of video segments that are arranged in order and are included in the historical feature queue and are located before the target video segment, to obtain a second feature vector corresponding to the first feature vector, and second historical feature information corresponding to the first historical feature information of each video segment in the k video segments; Determine a memory feature vector according to the second historical feature information of each video segment in the k video segments, wherein the memory feature vector includes short-term historical feature information and long-term historical feature information; the short-term historical feature information is determined based on the second historical feature information of n video segments adjacent to the target video segment in the historical feature queue, and the long-term historical feature information is determined based on the second historical feature information of m video segments in the historical feature queue except the n video segments adjacent to the target video segment, where k, n and m are positive integers, and m+n=k; The experimental process detection result of the target video segment is determined by a trained real-time process detection unit according to the second feature vector and the memory feature vector of the target video segment, and the experimental process detection result includes at least one process category of a variable gravity particle experiment, and the start time and end time corresponding to each process category.

2. The method according to claim 1, characterized in that The method further comprises: Based on a multi-layer perceptron, a projection operation is performed on the experimental process detection result of the target video segment to obtain a projection feature vector corresponding to the target video segment; Determine first historical feature information corresponding to the target video segment according to the projection feature vector and the first feature vector corresponding to the target video segment; Updating the historical feature queue according to the first historical feature information corresponding to the target video segment; Among them, the projection feature vector corresponding to the target video clip The formula for determining is: ； ； is the experimental process detection result of the target video segment, is the first feature vector corresponding to the target video segment, The first historical feature information corresponding to the target video segment.

3. The method according to claim 2, characterized in that The updating of the historical feature queue according to the first historical feature information corresponding to the target video segment includes: When the target video segment is the t-th of the multiple consecutive video segments of the variable gravity particle experiment, and t is less than or equal to k, adding the first historical feature information corresponding to the target video segment to the historical feature queue; When the target video segment is the tth of multiple consecutive video segments of the variable gravity particle experiment, and t is greater than k, the first historical feature information corresponding to the video segment ranked first in the historical feature queue is deleted, and the first historical feature information corresponding to the target video segment is added to the historical feature queue.

4. The method according to claim 3, characterized in that The step of determining the memory feature vector according to the second historical feature information of each video segment in the k video segments includes: splicing the second historical feature information of n video segments adjacent to the target video segment to obtain short-term historical feature information; Divide the second historical feature information of m video clips in the historical feature queue except the n video clips adjacent to the target video clip into x information groups, each information group includes the second historical feature information of y video clips, wherein x and y are positive integers, and x*y=m; Based on a compression aggregation network, the second historical feature information of the y video clips included in each information group is compressed and aggregated to obtain a compressed aggregation feature corresponding to each information group, wherein the compression aggregation network is a neural network structure based on Taylor expansion and Kolmogorov-Arnold representation theorem; The compressed aggregate features corresponding to each information group are concatenated to obtain long-term historical feature information; The short-term historical feature information and the long-term historical feature information are concatenated to obtain a memory feature vector.

5. The method according to claim 4, characterized in that The compression aggregation network is used to: Based on Taylor polynomials, feature transformation is performed on the second historical feature information of each information group including y video clips to obtain a transformation feature corresponding to each second historical feature information; Aggregation processing is performed on the transformed features corresponding to the second historical feature information included in each information group to obtain compressed aggregated features corresponding to each information group.

6. The method according to claim 5, characterized in that The trained real-time process detection unit includes a plurality of serially connected Transformer Decoder units, wherein the plurality of serially connected Transformer Decoder units are used to output a processing result according to an input query vector, a key vector, and a value vector; Experimental process detection results of the target video clip The formula for determining is: ； ； ； ； in, is the activation function; The processing result output by the Transformer Decoder unit; is the weight for projecting the processing results output by the Transformer Decoder unit into the category space, and c is the number of process categories; is the query vector, K is the key vector, and V is the value vector; is a second feature vector of the target video segment; is the memory feature vector of the target video segment.

7. The method according to claim 6, characterized in that Before determining the experimental process detection result of the target video segment by a real-time process detection unit according to the first feature vector and the memory feature vector of the target video segment, the method further includes: Acquire a training sample set of a variable gravity particle experiment, wherein the training sample set includes a plurality of training samples, each training sample includes a training video segment and an experimental process detection result corresponding to the training video segment; Construct the target loss function; Based on the target loss function, the real-time process detection unit is iteratively trained to obtain a trained real-time process detection unit.

8. The method according to claim 7, characterized in that The objective loss function for: ； ； ； in, is the balancing factor for class i in each batch, is the predicted probability that the number of video frames corresponding to time t is process category i; yes Moments belong to the process category The true label of is the balance factor; is the total number of process instances in each batch, is the mean statistic based on logarithmic transformation, is the process category in each batch The number of process instances.

9. A real-time detection system for variable gravity particle experiment process based on historical information, characterized in that: The system comprises: A video acquisition unit, used for acquiring a target video segment of a variable gravity particle experiment, wherein the target video segment includes a continuous plurality of frames of images, and the target video segment is one of a plurality of continuous video segments of the variable gravity particle experiment; A feature extraction unit, configured to determine a first feature vector corresponding to the target video segment; a position encoding unit, configured to perform a position encoding operation on the first feature vector and first historical feature information of k video segments in the plurality of video segments that are arranged in order and are included in the historical feature queue and are located before the target video segment, to obtain a second feature vector corresponding to the first feature vector, and second historical feature information corresponding to the first historical feature information of each video segment in the k video segments; a vector determination unit, configured to determine a memory feature vector according to the second historical feature information of each video segment in the k video segments, wherein the memory feature vector includes short-term historical feature information and long-term historical feature information; the short-term historical feature information is determined based on the second historical feature information of n video segments adjacent to the target video segment in the historical feature queue, and the long-term historical feature information is determined based on the second historical feature information of m video segments in the historical feature queue except the n video segments adjacent to the target video segment, where k, n and m are positive integers and m+n=k; A process detection unit is used to determine the experimental process detection result of the target video segment according to the second feature vector and the memory feature vector of the target video segment through a trained real-time process detection unit, wherein the experimental process detection result includes at least one process category of a variable gravity particle experiment, and the start time and end time corresponding to each process category.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the real-time detection method of the variable gravity particle experimental process based on historical information as described in any one of claims 1-8.

Images