A video driving device control method and system based on contribution frame interval verification

Abstract

This application discloses a video-driven device control method and system based on contribution frame interval verification, belonging to the field of video data processing and device linkage control technology. The method includes: acquiring a video frame sequence and playback timestamp; determining the contribution frame interval actually involved in motion feature calculation and the target scene confidence of each frame for the target control moment; determining the interval gating result based on the effective scene coverage ratio and the scene change state at the interval boundary; filtering effective contribution frames when output is allowed, calculating the target motion features of the target area relative to the background area and performing time normalization based on the playback timestamp to obtain the verification motion value; setting the verification motion value to a safe value when output is prohibited; generating and outputting device control curves based on the verification motion values ​​and playback timestamps of continuous target control moments. This application can reduce the probability of invalid image motion components entering the device control result and improve the timing matching between the video scene state and the device control output.

Description

Technical Field

[0001] This application belongs to the field of video data processing and equipment linkage control technology, specifically a video-driven equipment control method and system based on contribution frame interval verification. Background Technology

[0002] With the development of video content analysis and device linkage control technologies, device control data that changes with the video playback process can be generated based on the target motion state in the video frame, so that the actions of the controlled device correspond to the video content. In the existing video-driven device control process, a scene judgment model is usually used first to determine whether the current video frame belongs to a target scene that allows device linkage. Then, motion features are calculated based on the image changes between the current video frame and its adjacent video frames, and the motion features are converted into corresponding device control values.

[0003] However, the scope of video data used for scene validity assessment and motion feature calculation is not entirely consistent. Scene validity assessment is typically made for a single current video frame, while a motion feature needs to be determined by changes between two or more temporally consecutive video frames. Therefore, the fact that the current video frame meets the scene admission criteria does not necessarily mean that all other video frames actually involved in motion feature calculation are in the same valid scene state.

[0004] Especially at times when the target scene enters or exits, or when the camera state changes, multiple video frames actually involved in a single motion feature calculation may span different scene states. In this case, even if some video frames already have a high confidence level in the target scene, the overall displacement of the image generated by other video frames may still be included in the motion analysis results, causing image changes not caused by the target object's movement to further enter the device control data. This problem cannot be solved simply by improving the accuracy of single-frame scene classification; its root cause lies in the inconsistency between the object used to determine the validity of the control output and the video frame object actually used to generate that control output.

[0005] Furthermore, when some video frames cannot continue to participate in motion calculations due to unmet scene conditions, the actual number of effective video frames used to calculate motion features and the corresponding time spans at different control moments may differ. If control values ​​are still generated directly based on the original cumulative motion results, the motion results obtained under different effective time spans will be at different time scales, which can easily lead to inconsistencies in the control quantities corresponding to adjacent control moments.

[0006] Therefore, the technical problem that existing video-driven device control technology needs to solve is: how to match the scene validity judgment of the control output with the range of video frames that actually participate in the motion feature calculation when the control output is generated, and to keep the time scale of the motion result consistent when the actual effective motion calculation time changes, so as to reduce the impact of changes in non-target scene images on the device control result. Summary of the Invention

[0007] In existing video-driven device control processes, scene validity judgment is typically based on a single video frame, while motion feature calculation relies on multiple consecutive video frames. This leads to inconsistencies between the scope of scene validity judgment and the actual range of video frames involved in motion feature calculation. Furthermore, when the number of effective contributing frames or the effective time span changes, it is difficult to maintain a uniform time scale for motion results at different control moments. This application provides a video-driven device control method and system based on contribution frame interval verification. By determining the contribution frame interval actually involved in motion feature calculation for the target control moment, scene validity verification, motion feature calculation, and time normalization are performed based on the contribution frame interval. This reduces the probability of non-target scene image changes entering the device control results and improves the temporal matching between video scene state, motion analysis results, and device control output.

[0008] To achieve the above objectives, this application adopts the following technical solution.

[0009] In a first aspect, this application provides a video-driven device control method based on contribution frame interval verification, comprising: acquiring a video frame sequence and the playback timestamp corresponding to each video frame in the video frame sequence; for a target control moment, determining the contribution frame interval participating in motion feature calculation from the video frame sequence, and acquiring the target scene confidence of each video frame within the contribution frame interval; determining the effective scene coverage ratio of the contribution frame interval based on each target scene confidence, and determining the interval gating result of the contribution frame interval in combination with the scene change state of the interval boundary video frames of the contribution frame interval; when the interval gating result is allowed to output, filtering effective contribution frames that meet the scene admission conditions from the contribution frame interval, determining the target motion features of the target region relative to the background region based on the effective contribution frames, and performing time normalization on the target motion features based on the playback timestamp corresponding to the effective contribution frames to obtain the verification motion value of the target control moment; when the interval gating result is prohibited to output, setting the verification motion value of the target control moment to a safe value; generating and outputting a device control curve based on the verification motion value corresponding to consecutive target control moments and the playback timestamp.

[0010] The above technical solution no longer uses a single video frame corresponding to the target control moment as the criterion for judging the effectiveness of the control output. Instead, it uses multiple video frames that actually participate in the motion feature calculation at the target control moment as a contribution frame interval for verification, so that the scope of scene validity judgment can correspond to the range of video frames that the motion features actually depend on. At the same time, the effective scene coverage ratio and the scene change status of the interval boundary video frames are used to form an interval gating. After the output is allowed, the target motion features formed by the effective contribution frames are time-normalized according to the actual playback time. This ensures that the motion results corresponding to different numbers of effective video frames and different effective time spans are at a unified control time scale, thereby reducing the occurrence of scene switching, black screen, occlusion, or other invalid image changes in the device control results.

[0011] In one possible implementation, determining the contribution frame interval for the target control time and obtaining the target scene confidence of each video frame within the contribution frame interval includes: calculating the range of preceding and following video frames called based on motion features to determine the time support range corresponding to the target control time; reading video frames whose playback timestamps are within the time support range and actually participate in motion feature calculation from the video frame sequence, and arranging them according to the playback timestamps to form the contribution frame interval; and performing target scene judgment on each video frame within the contribution frame interval to obtain the target scene confidence of each video frame.

[0012] This implementation method calculates the range of video frames that actually need to be called based on motion features to form a contribution frame interval, instead of performing scene verification based on a single target control moment or a fixed single frame. This ensures that the target scene confidence acquisition object is consistent with the video frame object actually used for motion feature calculation, providing a data basis for the overall scene validity judgment of the contribution frame interval.

[0013] In one possible implementation, determining the effective scene coverage ratio based on the confidence level of each target scene includes: comparing the confidence level of each target scene with a scene admission threshold, counting the number of video frames whose confidence level of the target scene reaches the scene admission threshold, and determining the effective scene coverage ratio as the ratio between the number of video frames and the total number of video frames in the contribution frame interval.

[0014] This implementation converts the number of video frames that meet the scene admission criteria within the contribution frame interval into a proportion relative to the total number of video frames in the interval. This allows contribution frame intervals with different numbers of video frames to be evaluated using a unified metric to assess the coverage of effective target scene video frames, providing a comparable quantitative basis for determining the interval gating results.

[0015] In one possible implementation, the interval gating result is determined by combining the scene change status of the interval boundary video frames of the contributing frame interval, including: determining the scene change status of the two interval boundaries based on the target scene confidence change between the starting boundary video frame and its adjacent video frames within the contributing frame interval, and the target scene confidence change between the ending boundary video frame and its adjacent video frames within the contributing frame interval; when the effective scene coverage ratio reaches a coverage ratio threshold, both the starting boundary video frame and the ending boundary video frame meet the scene stability condition, and there is at least one set of adjacent video frames in the video frame sequence within the contributing frame interval that both meet the scene admission condition, the interval gating result is determined to be allowed output; otherwise, the interval gating result is determined to be prohibited output.

[0016] This implementation method introduces scene change states at the start and end boundaries of the contribution frame interval in addition to the effective scene coverage ratio. It also requires that there are actually adjacent effective video frames within the interval that can form the motion analysis object. This makes the interval gating not only reflect the proportion of the number of target scene video frames within the interval, but also able to identify whether the contribution frame interval crosses the scene entry, scene exit, or shot change boundary. This reduces the probability of incorrectly opening the control output based solely on the majority of video frames within the interval meeting the scene admission conditions.

[0017] In one possible implementation, determining the target motion feature based on the effective contribution frames includes: identifying video frames whose target scene confidence reaches the scene admission threshold as effective contribution frames; forming effective frame pairs from two adjacent video frames in the video frame sequence that are both effective contribution frames; and identifying the video frame with the earlier playback timestamp in each effective frame pair as a reference video frame; determining a target region and a background region for the reference video frame of each effective frame pair; calculating the target region motion amount of the target region and the background region motion amount of the background region, respectively; and determining the difference between the target region motion amount and the background region motion amount as the target motion feature of the corresponding effective frame pair.

[0018] This implementation method utilizes only consecutive video frames in the original video frame sequence that meet the scene admission criteria to form effective frame pairs, thus avoiding the need to re-establish motion relationships across rejected video frames. Furthermore, by forming target motion features based on the difference between the motion amount in the target area and the motion amount in the background area, the common motion of the target area and the screen can be distinguished, thereby reducing the impact of background motion components such as overall camera panning and shaking on the motion representation of the target object.

[0019] In one possible implementation, time normalization of the target motion features includes: determining the inter-frame time of each effective frame pair based on the playback timestamps corresponding to the two effective contribution frames in each effective frame pair, and adding the inter-frame times to obtain the effective time length; accumulating the target motion features corresponding to each effective frame pair, and converting the accumulated target motion amount to a motion amount scale corresponding to a preset control time length based on the effective time length, and determining the converted target motion amount as the verification motion value at the target control moment.

[0020] This implementation method uses the playback timestamps corresponding to the effective contribution frames calculated from the actual motion characteristics to determine the effective time length, and converts the cumulative target motion amount to a unified preset control time length. This allows different effective time spans caused by video frame filtering, variable video frame rates, or different numbers of effective frame pairs within the contribution frame interval to be unified to the same control time scale, thereby improving the comparability between the verification motion values ​​corresponding to continuous target control moments.

[0021] In one possible implementation, setting the verification motion value to a safe value includes: determining a safe value with the same data format as the verification motion value, based on the data format of the verification motion value, to indicate that no target motion drive is generated at the target control moment; when the interval gating result is prohibited from output, not using the motion analysis result corresponding to the contribution frame interval as the basis for generating the device control curve, and setting the verification motion value corresponding to the target control moment to the safe value.

[0022] This implementation method prevents the corresponding motion analysis result from continuing into the device control curve generation process when the contribution frame interval does not meet the output conditions. Instead, it replaces the motion result at the target control moment with a safety value in the same data format as the normal verification motion value. This allows both the allowed and prohibited output paths to provide the same data level input to the subsequent device control value mapping process, thereby reducing the complexity of subsequent control processing caused by abnormal branches using different data structures.

[0023] In one possible implementation, generating and outputting the device control curve includes: converting the verification motion value corresponding to each target control moment into a device control value according to a pre-established mapping relationship between verification motion values ​​and device control values; combining the playback timestamp corresponding to each target control moment with the corresponding device control value to form multiple control points, and arranging the multiple control points according to the playback timestamp to form the device control curve; during video playback, reading and outputting the corresponding device control value in the device control curve according to the current playback time.

[0024] This implementation method converts the verified motion values ​​after contribution frame interval verification into device control values ​​that the controlled device can use. The device control values ​​are then located on the video playback timeline using the playback timestamp, establishing a time correspondence between each control point and the video playback process. This allows the controlled device to read the corresponding device control values ​​according to the actual playback progress of the video, thus achieving time-series linkage between video motion analysis results and device actions.

[0025] In one possible implementation, the determination of the coverage ratio threshold and the boundary change threshold for forming the scene stability condition includes: obtaining a first sample contribution frame interval pre-labeled as allowed output and a second sample contribution frame interval pre-labeled as prohibited output; setting multiple candidate coverage ratio thresholds according to the effective scene coverage ratio of each sample contribution frame interval; counting the number of interval misjudgments corresponding to each candidate coverage ratio threshold; and determining the candidate coverage ratio threshold with the smallest number of interval misjudgments as the coverage ratio threshold; obtaining a first boundary sample pre-labeled as a scene stable state and a second boundary sample pre-labeled as a scene changing state; setting multiple candidate boundary change thresholds according to the target scene confidence difference between adjacent video frames in each boundary sample; counting the number of boundary state misjudgments corresponding to each candidate boundary change threshold; and determining the candidate boundary change threshold with the smallest number of boundary state misjudgments as the boundary change threshold.

[0026] In this implementation method, the coverage ratio threshold is determined based on the historical annotation results of whether the entire contribution frame interval is suitable for output, while the boundary change threshold is determined based on the historical annotation results of whether the interval boundary is in a scene change state. This allows the two thresholds to be determined by sampling for different judgment objects, and the corresponding candidate thresholds are filtered by the number of misjudgments, thereby reducing the interval gating judgment deviation caused by setting thresholds with arbitrary fixed values.

[0027] Secondly, this application provides a video-driven device control system based on contribution frame interval verification, implementing the aforementioned video-driven device control method based on contribution frame interval verification, including a video acquisition module, a contribution interval determination module, an interval gating module, a motion verification module, and a curve output module; the video acquisition module is used to acquire a video frame sequence and the playback timestamp corresponding to each video frame in the video frame sequence; the contribution interval determination module is used to determine the contribution frame interval participating in motion feature calculation from the video frame sequence for a target control time, and to acquire the target scene confidence of each video frame within the contribution frame interval; the interval gating module is used to determine the effective scene coverage ratio of the contribution frame interval based on the confidence of each target scene, and in conjunction with the contribution frame... The scene change state of the video frames at the interval boundaries is used to determine the interval gating result of the contributing frame interval. The motion verification module is used to filter valid contributing frames that meet the scene admission conditions from the contributing frame interval when the interval gating result is allowed to output. Based on the valid contributing frames, the target motion features of the target area relative to the background area are determined, and the target motion features are time-normalized according to the playback timestamp corresponding to the valid contributing frames to obtain the verification motion value of the target control moment. When the interval gating result is prohibited to output, the verification motion value of the target control moment is set to a safe value. The curve output module is used to generate and output the device control curve based on the verification motion value corresponding to the continuous target control moments and the playback timestamp.

[0028] Compared with related technologies, this application has the following advantages: By determining the contribution frame intervals for actual motion feature calculations at the target control time, and forming an effective scene coverage ratio based on the target scene confidence of each video frame within the interval, and combining the scene change status of the video frames at the interval boundaries to determine the interval gating result, the processing object for scene validity judgment can correspond to the range of video frames that the motion features actually depend on at the current control time. This reduces the probability of non-target motions such as scene switching entering the control result based solely on opening the control output based on a single video frame. After the interval gating allows the output, effective contribution frames that meet the scene admission conditions are selected, and the target is determined based on the motion difference between the target area and the background area. Motion features enable the differentiation between common motion in the frame and motion in the target area during the motion feature formation process, further reducing motion components unrelated to the target object's motion. Simultaneously, the target motion features are time-normalized based on the playback timestamps corresponding to the effective contribution frames, transforming motion results corresponding to different numbers of effective video frames and different effective time spans to a unified control time scale. When output is prohibited by interval gating, the verification motion value is set to a safe value. This allows the verification motion values ​​formed at continuous target control moments to generate device control curves at a unified data level, improving the temporal matching between video scene status, motion analysis results, and device control output. Attached Figure Description

[0029] Figure 1 This application provides a schematic diagram of a video driving device control application scenario. Figure 2 A flowchart of a video driving device control method based on contribution frame interval verification is provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the correspondence between target control time and contribution frame interval provided in an embodiment of this application; Figure 4 A flowchart for determining the effective scene coverage ratio of a contribution frame interval is provided in an embodiment of this application; Figure 5 This application provides a schematic diagram of scene change status and interval gating judgment for an interval boundary video frame. Figure 6 This is a schematic diagram illustrating the determination of motion features between a target region and a background region, provided in an embodiment of this application. Figure 7 This application provides a schematic diagram of effective contribution frame time normalization in its embodiments. Figure 8 This application provides a schematic diagram of a device control curve generation method. Figure 9 This is a schematic diagram of a video driving device control system structure based on contribution frame interval verification, provided as an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and technical effects of the embodiments of this application clearer, the embodiments of this application will be further described below with reference to the accompanying drawings. The following embodiments are used to illustrate this application and are not intended to limit the scope of protection of this application.

[0031] like Figure 1 As shown, the method of this application can be executed by a video processing device. The video processing device obtains a video frame sequence and playback timestamp from a video file or video playback data stream, performs contribution frame interval verification and verification motion value calculation for multiple target control moments in the video playback timeline, and finally sends the resulting device control curve to the controlled device according to the video playback time.

[0032] A video frame sequence is a series of video frames arranged in the order in which the video is played.

[0033] Playback timestamps are used to indicate the time position of the corresponding video frame in the video playback timeline.

[0034] The target control moment is the time position in the equipment control curve where a verification motion value needs to be generated. Each target control moment corresponds to a timestamp on the video playback timeline.

[0035] In one implementation, when a video frame is used as the sampling reference for device control, the playback timestamp of the corresponding video frame is determined as the target control time.

[0036] In another implementation, when the device control cycle is out of sync with the video frame cycle, multiple target control moments are sequentially determined on the video playback timeline according to the device control cycle.

[0037] The continuous target control moments refer to multiple target control moments arranged sequentially according to the video playback timeline to form the device control curve.

[0038] The contribution frame interval is the range of video frames that actually participate in motion analysis in order to form the motion characteristics corresponding to a target control moment.

[0039] When processing is performed sequentially at consecutive target control moments, the contribution frame interval can be represented as a sliding window that moves along the video playback time axis with the target control moment. Contribution frame intervals corresponding to adjacent target control moments are allowed to contain some identical video frames. Each contribution frame interval corresponds to a target control moment. The interval gating result determined for the contribution frame interval is used to determine the verification motion value corresponding to that target control moment, and is not directly used as the unified gating result for each video frame within the contribution frame interval or for other target control moments.

[0040] The target scene confidence score is used to represent the probability that the corresponding video frame belongs to the target scene that allows the device to perform linkage control.

[0041] A valid contribution frame is a video frame within the contribution frame interval that meets the scene admission criteria and can continue to participate in motion analysis.

[0042] Target motion features are used to characterize the motion of the target area relative to the background area.

[0043] The verification motion value is the motion quantity formed after verification of the contribution frame interval, screening of effective contribution frames and time normalization, and is used as the input for subsequent device control value mapping.

[0044] like Figure 2 As shown, a video driving device control method based on contribution frame interval verification according to this application includes steps S101 to S106.

[0045] Step S101: Obtain the video frame sequence and the playback timestamp corresponding to each video frame in the video frame sequence. The detailed implementation steps of step S101 include: Step S1011: Obtain the video frame sequence. Specifically, decode the video data to be controlled by the device linkage and obtain multiple video frames according to the video playback order.

[0046] The video data can be read from locally stored video files or from video data streams output from the video playback interface.

[0047] The first Each video frame is denoted as The video frame sequence is then represented as: .

[0048] in, This represents the video frame sequence; Represents the first frame in the video frame sequence. One video frame; Indicates the video frame number; This represents the total number of video frames in the video frame sequence, and .

[0049] Therefore, the input video data is decoded and arranged according to the playback order to obtain the video frame sequence.

[0050] Step S1012: Obtain the playback timestamp corresponding to each video frame in the video frame sequence. Specifically, when the video data itself records the playback time information of each video frame, the corresponding playback time information is read synchronously when decoding the video frame, and the playback time information is used as the playback timestamp of the video frame.

[0051] When video data is played at a fixed frame rate and frame-by-frame timestamps are not directly recorded, the following formula is used to determine the [number of frames]. The playback timestamps corresponding to each video frame: .

[0052] in, Represents video frames Playback timestamp; This represents the playback timestamp of the first video frame in the video frame sequence; Indicates the video frame rate, and .

[0053] When the video uses a variable frame rate, the playback time information of each frame is read from the video data record, and the playback timestamp is not calculated using a fixed video frame rate.

[0054] Therefore, a corresponding video playback time position is established for each video frame in the video frame sequence obtained in step S1011, and the playback timestamp corresponding to each video frame is obtained.

[0055] In step S101, a video frame sequence and a playback timestamp corresponding to each video frame are obtained. In step S102, the video frame sequence and playback timestamp are used to determine the contribution frame interval corresponding to the target control time. The playback timestamp is also used for time normalization in step S104 and for time positioning of the device control curve in step S106.

[0056] For example, taking a video with a fixed frame rate of 30 frames per second as an example, assuming the playback timestamp of the first video frame is 0 seconds, the video processing device will number the decoded video frames sequentially according to the playback order. , ...When frames 99, 100, and 101 are read, their playback timestamps are determined to be approximately 3.267 seconds, 3.300 seconds, and 3.333 seconds, respectively, based on the video frame sequence number and video frame rate. Therefore, the video processing device not only saves the image data corresponding to frames 99 to 101, but also saves the playback timestamp corresponding to each video frame. Subsequently, when using 3.300 seconds as a target control moment, the range of video frames required for motion feature calculation at that target control moment can be directly determined based on the aforementioned video frame sequence and playback timestamps, thus allowing the video frame sequence and playback timestamps obtained in step S101 to be input into step S102.

[0057] Step S102: For the target control time, determine the contributing frame intervals involved in motion feature calculation from the video frame sequence, and obtain the target scene confidence of each video frame within the contributing frame intervals. The detailed implementation steps of step S102 include: Step S1021: For the target control time, determine the contributing frame intervals involved in motion feature calculation from the video frame sequence; such as... Figure 3 As shown, the contribution frame interval corresponding to a target control moment covers multiple video frames actually needed to form the motion features of that target control moment, rather than just including a single video frame corresponding to the target control moment.

[0058] Specifically, before performing motion analysis, the range of preceding and following video frames required to form a motion feature is determined based on the current motion feature calculation method.

[0059] For example, when the current motion feature is formed by the video frame corresponding to the target control time, its preceding video frame, and its following video frame, it is necessary to call a preceding video frame, a video frame corresponding to the target control time, and a following video frame.

[0060] The first Each target control moment is denoted as .

[0061] Based on the preceding and following time lengths required to be covered by the current motion feature calculation method, determine: .

[0062] in, Indicates the first The time support range corresponding to each target control moment; This indicates the length of time that needs to be covered before the target control moment for motion feature calculation; This indicates the length of time that needs to be covered after the target control moment for motion feature calculation; and .

[0063] Read video frames whose playback timestamps are within the time support range and are actually used in the current motion feature calculation from the video frame sequence obtained in step S101, and form: .

[0064] in, Indicates the target control time The corresponding contribution frame range.

[0065] When the motion feature calculation method directly calls a fixed number of preceding and following video frames according to the video frame sequence number, it can also directly read a specified number of video frames forward and backward according to the corresponding video frame sequence number to form the contribution frame interval, and use the playback timestamps corresponding to the read video frames to determine its actual time span.

[0066] When the target control time is located at the beginning or end of the video, causing the video frames required by the current motion feature calculation method to be unable to be obtained completely, the corresponding contribution frame interval is determined as the contribution frame insufficiency interval. The contribution frame insufficiency interval directly corresponds to the interval gating result that is prohibited from output in step S103.

[0067] Therefore, based on the video frame sequence, playback timestamp, target control time, and motion features obtained in step S101, the required video frame range is calculated, and the contribution frame interval is obtained.

[0068] In one implementation, when generating device control values ​​sequentially according to multiple consecutive target control times, the target control times are used as a sliding reference, causing the contributing frame intervals to move sequentially along the video playback time axis with the target control times. For any target control time, subsequent scene verification and motion analysis are performed only using the video frames in the contributing frame interval corresponding to the current target control time, and an interval gating result is obtained for the current target control time. When processing the next target control time, the corresponding contributing frame interval is re-determined based on the next target control time, and the corresponding interval gating result is re-determined. Therefore, even if there are partially overlapping video frames between two adjacent contributing frame intervals, the interval gating result of each contributing frame interval still corresponds to its respective target control time.

[0069] Step S1022: Obtain the target scene confidence of each video frame in the contribution frame interval. Specifically, each video frame in the contribution frame interval is input into a pre-obtained scene judgment model, the probability output by the scene judgment model for the target scene category is read, and the probability is determined as the target scene confidence of the corresponding video frame.

[0070] The scene judgment model can be implemented using an existing scene classification model with probability output capability. The training samples for the scene judgment model are derived from historical video data. Video frames in the training samples are pre-labeled with target scene tags or non-target scene tags. Specifically, the scene judgment model uses historical video frames pre-labeled with target scene tags and non-target scene tags as training samples. The tag value of target scene video frames is set to 1, and the tag value of non-target scene video frames is set to 0. The training samples are divided into a training set and a validation set. During training, video frames from the training set are input into the scene judgment model, the target scene prediction probability output by the model is read, and the training loss between the prediction probability and the corresponding scene tag is calculated using binary cross-entropy. The model parameters are then iteratively updated based on the training loss. After each round of model parameter updates, video frames from the validation set are input into the current scene judgment model, and the same calculation is performed. The corresponding validation loss is calculated, and the validation loss of the current round is compared with the validation loss of the previous round. When the change in validation loss for multiple consecutive training rounds is not greater than the preset loss convergence threshold, the scene judgment model is determined to have reached the convergence condition and the parameter update is stopped. The preset loss convergence threshold and the number of consecutive training rounds are determined based on the changes in validation loss during historical training. After training, the model parameters corresponding to the convergence condition are saved to obtain the scene judgment model. In step S1022, each video frame in the contribution frame interval is input into the scene judgment model, and the corresponding target scene prediction probability is read as the target scene confidence, so that step S103 can determine the effective scene coverage ratio and the scene change status of the video frames at the interval boundary.

[0071] for: .

[0072] We obtained the following respectively: .

[0073] in, Indicates the target control time The set of target scene confidence scores corresponding to the contribution frame interval; Represents video frames The corresponding target scenario confidence level; and: .

[0074] Therefore, using the contribution frame interval obtained in step S1021 as the processing object, the target scene confidence level corresponding to each video frame within the contribution frame interval is obtained.

[0075] In one implementation, steps S101 to S106 corresponding to the entire video frame sequence can be completed before the video is actually played, thus generating a complete device control curve in advance.

[0076] In another implementation, when the contribution frame interval corresponding to the target control time includes video frames after the target control time, the video frames are cached until the last video frame that needs to be called in the current contribution frame interval is obtained, and then the verification motion value corresponding to the target control time is calculated.

[0077] In step S102, the contribution frame interval corresponding to the target control time and the target scene confidence of each video frame within the contribution frame interval are obtained. The contribution frame interval and the target scene confidence are then used in step S103 to determine the effective scene coverage ratio and the scene change status of the video frames at the interval boundaries.

[0078] For example, continuing with the target control time of 3.300 seconds, the current motion feature calculation method requires the use of the video frame corresponding to the target control time, the preceding video frame, and the following video frame. Therefore, the video processing device, based on the video frame sequence and playback timestamp obtained in step S101, determines frames 99, 100, and 101 as the contributing frame interval corresponding to the target control time. Subsequently, frames 99, 100, and 101 are input into the trained scene judgment model, respectively. Assuming the obtained target scene confidence scores are 0.93, 0.96, and 0.94, respectively, a target scene confidence score set corresponding to the contributing frame interval is formed. Thus, step S102 expands the processing scope from only performing scene judgment on frame 100 to all video frames actually involved in the motion feature calculation for the target control time, and provides the contributing frame interval and the target scene confidence scores corresponding to each video frame to step S103.

[0079] Step S103: Determine the effective scene coverage ratio of the contributing frame interval based on the confidence level of each target scene, and determine the interval gating result of the contributing frame interval by combining the scene change status of the interval boundary video frames of the contributing frame interval.

[0080] The detailed implementation steps of step S103 include: Step S1031: Determine the effective scene coverage ratio of the contributing frame interval based on the confidence level of each target scene, such as... Figure 4 As shown, the target scene confidence of each video frame in the contribution frame interval is compared with the scene admission threshold, and the effective scene coverage ratio is determined according to the proportion of video frames that reach the scene admission threshold in the contribution frame interval.

[0081] Let the scene admission threshold be denoted as .

[0082] For any video frame in the contributing frame interval The corresponding scene admission judgment value is determined according to the following formula: .

[0083] in, Represents video frames The corresponding scene admission criteria; Represents video frames Meets the scene access requirements; Represents video frames The scene access conditions are not met.

[0084] for: .

[0085] The total number of video frames within the contributing frame interval is: .

[0086] in, .

[0087] The effective scene coverage ratio is determined according to the following formula: .

[0088] in, Indicates the first The effective scene coverage ratio of each contributing frame interval, and: .

[0089] The scene access threshold Determined based on pre-labeled historical target scene video frames and historical non-target scene video frames.

[0090] Specifically, historical target scene video frames and historical non-target scene video frames are input into the scene judgment model to obtain the corresponding target scene confidence scores; the confidence scores of different target scenes and the median value between two adjacent different target scene confidence scores are used as the candidate scene admission threshold.

[0091] For each candidate scene admission threshold, the number of first misjudgments where non-target scene video frames are judged to meet the scene admission conditions and the number of second misjudgments where target scene video frames are judged not to meet the scene admission conditions are counted. The first misjudgment number and the second misjudgment number are added together to obtain the total number of misjudgments for the corresponding candidate scene admission threshold.

[0092] The candidate scene admission threshold with the smallest total number of misjudgments is determined as the scene admission threshold; when at least two candidate scene admission thresholds correspond to the same minimum total number of misjudgments, the candidate scene admission threshold with the larger value is determined as the scene admission threshold.

[0093] Therefore, based on the target scene confidence obtained in step S102, the effective scene coverage ratio of the contributing frame interval is obtained.

[0094] Step S1032: Determine the scene change status of the video frames at the interval boundaries of the contributing frame interval.

[0095] like Figure 5 As shown, after obtaining the effective scene coverage ratio, it is determined whether the start and end boundaries of the contributing frame interval are at the scene state change position.

[0096] Specifically, for a contributing frame interval that is not a contributing frame insufficiency interval and contains at least two video frames, the video frame with the earliest timestamp will be played. The video frame with the latest timestamp will be played if the starting boundary video frame is determined. The video frame was identified as the end boundary.

[0097] Determined based on video frames within the contribution frame interval: .

[0098] as well as: .

[0099] in, This represents the change in confidence level at the starting boundary of the contributing frame interval; The change in confidence level at the end boundary of the contributing frame interval is shown. and These represent the target scene confidence scores corresponding to the start boundary video frame and the end boundary video frame, respectively. This represents the target scene confidence level corresponding to the video frame adjacent to the starting boundary video frame within the contribution frame interval; This represents the target scene confidence level corresponding to the video frame adjacent to the end boundary video frame within the contribution frame interval.

[0100] The preset boundary change threshold is denoted as... .

[0101] When both conditions are met: ,

[0102] as well as: .

[0103] If the scene change state of the starting boundary video frame is stable, then the scene change state of the starting boundary video frame is determined as a stable scene state; otherwise, the scene change state of the starting boundary video frame is determined as a scene change state.

[0104] When both conditions are met: , .

[0105] as well as: .

[0106] If the scene change state of the ending boundary video frame is determined to be a stable scene state, then the scene change state of the ending boundary video frame is determined to be a scene change state.

[0107] Therefore, the scene stability conditions specifically include: the video frame at the interval boundary and its adjacent video frames within the contributing frame interval both satisfy the scene admission conditions, and the absolute value of the difference in target scene confidence between the two is not greater than a preset boundary change threshold.

[0108] The preset boundary change threshold Boundary samples are determined based on historical videos. Specifically, first boundary samples pre-labeled as stable scene states and second boundary samples pre-labeled as changing scene states are obtained. The absolute value of the target scene confidence difference between two adjacent video frames in each boundary sample is calculated. The different differences and the median value between two adjacent different differences are determined as candidate boundary change thresholds.

[0109] For each candidate boundary change threshold, boundary samples with an absolute difference not greater than the candidate boundary change threshold are judged as stable scene states, and boundary samples with an absolute difference greater than the candidate boundary change threshold are judged as changing scene states. The number of misjudged boundary states where the judgment result is inconsistent with the sample label is also counted.

[0110] The candidate boundary change threshold with the smallest number of boundary state misjudgments is determined as the preset boundary change threshold.

[0111] When at least two candidate boundary change thresholds correspond to the same minimum number of misjudged boundary states, the candidate boundary change threshold with the smaller value is determined as the preset boundary change threshold.

[0112] Therefore, using the contribution frame interval and its target scene confidence obtained in step S102 as input, the scene change state corresponding to the starting boundary video frame and the ending boundary video frame is obtained.

[0113] Step S1033: Determine the interval gating result of the contribution frame interval. Specifically, according to the frame number in the video frame sequence, check the scene admission judgment value corresponding to two adjacent video frames in the contribution frame interval in turn.

[0114] When the scene admission judgment value corresponding to two adjacent video frames is 1, the two adjacent video frames are determined as a video frame pair that meets the motion analysis conditions.

[0115] When there is at least one pair of video frames in the contributing frame interval that meets the motion analysis conditions, the contributing frame interval is determined to meet the minimum motion analysis conditions.

[0116] The preset coverage ratio threshold is denoted as... .

[0117] When the contributing frame interval is not a contributing frame insufficiency interval and the effective scene coverage ratio satisfies: .

[0118] When both the starting boundary video frame and the ending boundary video frame satisfy the scene stability condition in step S1032, and the contributing frame interval satisfies the minimum motion analysis condition, the interval gating result of the contributing frame interval is determined as an allowed output.

[0119] In other cases, the interval gating result of the contributing frame interval is determined to be prohibited from output.

[0120] The interval gating result is expressed as: .

[0121] in, Indicates the first The interval gating result corresponding to each target control time.

[0122] The preset coverage ratio threshold Determined based on pre-labeled historical contribution frame interval samples.

[0123] Specifically, multiple first sample contribution frame intervals pre-labeled as allowed output and multiple second sample contribution frame intervals pre-labeled as prohibited output are obtained, and the effective scene coverage ratio of each sample contribution frame interval is calculated according to step S1031.

[0124] The coverage ratio of each different effective scene and the median value between two adjacent different effective scene coverage ratios are used as the candidate coverage ratio threshold.

[0125] For each candidate coverage ratio threshold, the sample contribution frame intervals with effective scene coverage ratios not less than the candidate coverage ratio threshold are judged as allowed output, and the sample contribution frame intervals with effective scene coverage ratios less than the candidate coverage ratio threshold are judged as prohibited output. The number of misjudged intervals where the judgment result is inconsistent with the corresponding sample label is counted.

[0126] The candidate coverage ratio threshold with the smallest number of misjudgments in an interval is determined as the preset coverage ratio threshold.

[0127] When at least two candidate coverage ratio thresholds correspond to the same minimum interval misclassification number, the candidate coverage ratio threshold with the larger value is determined as the preset coverage ratio threshold.

[0128] For example, when frames 99, 100, and 101 together form a contributing frame interval corresponding to a target control moment, and the target scene confidence of frame 100 is 0.92 and meets the scene admission criteria, while frames 99 and 101 do not meet the scene admission criteria, the effective scene coverage ratio of this contributing frame interval is: .

[0129] Meanwhile, frames 99 and 100 cannot form two video frame pairs that both meet the scene admission conditions, and frames 100 and 101 cannot form two video frame pairs that both meet the scene admission conditions. Therefore, the contribution frame interval does not meet the minimum motion analysis conditions, and the interval gating result is determined to be prohibited from output.

[0130] Step S103 yields a clear interval gating result; when the interval gating result is "output allowed", proceed to step S104; when the interval gating result is "output prohibited", proceed to step S105.

[0131] For example, continuing with the contribution frame interval consisting of frames 99, 100, and 101, assuming the target scene confidence scores for the three video frames are 0.93, 0.96, and 0.94 respectively, and the scene admission threshold is 0.80, then all three video frames meet the scene admission condition, and their corresponding scene admission judgment values ​​are all 1. Therefore, the effective scene coverage ratio is 3 / 3, or 1. Assuming the current preset coverage ratio threshold is 2 / 3, then the effective scene coverage ratio is not less than the preset coverage ratio threshold. Furthermore, the absolute value of the target scene confidence score difference between frame 99 and frame 100 is 0.03, and the absolute value of the target scene confidence score difference between frame 100 and frame 101 is 0.02. When the preset boundary change threshold is 0.10, both the starting boundary video frame and the ending boundary video frame meet the scene stability condition. Meanwhile, frames 99 and 100, as well as frames 100 and 101, can form video frame pairs that satisfy the motion analysis conditions. Therefore, this contributing frame interval meets the minimum motion analysis conditions. Thus, step S103 determines the interval gating result of this contributing frame interval as an allowed output, and provides the allowed output interval gating result and the corresponding contributing frame interval to step S104 for further filtering of valid contributing frames and calculation of target motion features.

[0132] Step S104: When the interval gating result is allowed output, select valid contribution frames that meet the scene admission conditions from the contribution frame interval, determine the target motion features of the target area relative to the background area based on the valid contribution frames, and perform time normalization on the target motion features based on the playback timestamp corresponding to the valid contribution frames to obtain the verification motion value of the target control time.

[0133] The detailed implementation steps of step S104 include: Step S1041: Filter valid contribution frames that meet the scene admission criteria from the contribution frame interval. Specifically, when step S103 yields: At that time, the target scene confidence level corresponding to each video frame in the contribution frame interval is read.

[0134] Will satisfy: The video frames were identified as valid contributing frames: .

[0135] in, Indicates the target control time The corresponding set of valid contribution frames.

[0136] The valid contributing frames are checked sequentially according to the frame number of the original video frame sequence, and two adjacent video frames in the video frame sequence that both belong to the set of valid contributing frames are combined into a valid frame pair.

[0137] All valid frame pairs are combined into a set of valid frame pairs: .

[0138] in, Indicates the target control time The corresponding set of valid frame pairs.

[0139] Since step S103 has already taken the requirement that the contributing frame interval meets the minimum motion analysis condition as a necessary condition for interval gating to allow output, therefore: .

[0140] Two valid contributing frames that do not meet the scene admission criteria and are separated by video frames in the original video frame sequence will not be combined into a valid frame pair, thereby avoiding the video time segment that was removed from the calculation of motion features again.

[0141] Thus, the set of effective contribution frames and effective frame pairs for motion analysis in step S1042 is obtained.

[0142] Step S1042: Determine the target motion features of the target region relative to the background region based on the effective contribution frames, such as... Figure 6 As shown, for each valid frame pair, the motion of the target region and the motion of the background region are determined respectively, and the target motion feature is formed based on the difference between the two. Specifically, before performing video motion analysis, non-empty target and background regions are established for the video to be processed.

[0143] In one implementation, the target region is determined by the region of interest recorded in the configuration file of the video processing task, and the background region is determined by a pre-configured pixel region outside the target region.

[0144] In another implementation, the target region can be determined by the target position record output by the existing video target detection or region segmentation process for the video frame. The target position record is saved together with the corresponding video frame before entering motion analysis.

[0145] For any valid frame pair ,in: .

[0146] Play the video frame with the earlier timestamp. The reference video frame is determined as the valid frame pair.

[0147] The set of pixel positions of the target object in the reference video frame is denoted as: .

[0148] The set of pixel positions in the reference video frame that do not overlap with the target area and are used to represent the common motion of the image is denoted as: .

[0149] in, Indicates reference video frame The target area in; Indicates reference video frame The background area in the text.

[0150] Both the target region and the background region are non-empty regions, that is: .

[0151] as well as: .

[0152] in, Indicates the number of pixels in the target area; This indicates the number of pixels in the background region. For valid frame pairs... Using existing dense optical flow calculation methods, based on video frames and video frames The image changes between them yielded: .

[0153] in, Indicates the pixel position in the reference video frame The corresponding two-dimensional optical flow vector; This indicates the pixel displacement in the horizontal direction at that pixel position; This indicates the vertical displacement of the pixel position.

[0154] When pixel coordinates are used to represent image positions, the units for both horizontal and vertical pixel displacements are pixels.

[0155] Determine the average motion vector of the target region based on the optical flow vector in the target region: .

[0156] in, Indicates a valid frame pair The corresponding average motion vector of the target area.

[0157] Determine the average motion vector of the background region based on the optical flow vector in the background region: .

[0158] in, Indicates a valid frame pair The corresponding average motion vector of the background region.

[0159] Determined based on the difference between the average motion vector of the target region and the average motion vector of the background region: .

[0160] in, Indicates a valid frame pair The corresponding target motion characteristics.

[0161] The vector direction of the target motion feature represents the motion direction of the target region relative to the background region, and the vector magnitude of the target motion feature represents the degree of motion of the target region relative to the background region. Subsequent steps directly use the target motion feature to perform cumulative summation and time normalization, without generating independent intermediate results of motion amplitude.

[0162] Therefore, for each valid frame pair obtained in step S1041, the target motion features of the corresponding target region relative to the background region are obtained respectively.

[0163] Step S1043: Perform time normalization on the target motion features based on the playback timestamps corresponding to the effective contribution frames to obtain the verification motion value at the target control moment, such as... Figure 7 As shown, when some video frames in the contribution frame interval do not enter the motion analysis because they do not meet the scene admission conditions, the effective time length of the actual participation in motion analysis at different target control times may be different. Therefore, the motion features of the corresponding targets are time-normalized according to the actual playback timestamp of each effective contribution frame.

[0164] For valid frame pairs Determined based on the playback timestamps corresponding to the two video frames: .

[0165] in, This indicates the inter-frame time corresponding to the valid frame pair.

[0166] because and Arranged in chronological order of playback time, therefore: .

[0167] Add the inter-frame times corresponding to all valid frame pairs: .

[0168] in, Indicates the target control time The corresponding effective time length.

[0169] because: .

[0170] And each valid frame pair satisfies: .

[0171] therefore: .

[0172] Add the target motion features corresponding to each valid frame pair: .

[0173] in, Indicates the target control time The corresponding cumulative target amount of exercise.

[0174] The uniform control time length used in the equipment control process is denoted as: .

[0175] The verification motion value corresponding to the target control moment is determined according to the following formula: .

[0176] in, Indicates the target control time The corresponding check motion value; This indicates the preset control time length.

[0177] When the equipment operates at a fixed control frequency When outputting device control values: .

[0178] in, The output frequency represents the device control value, and:

[0179] Therefore, the cumulative target motion obtained under different effective time lengths is converted to the same control time scale to obtain the verification motion value corresponding to the target control time.

[0180] In step S104, after the interval gating allows output, effective contribution frames and effective frame pairs are first obtained. Then, based on the effective frame pairs, the target motion features of the target area relative to the background area are obtained. The target motion features are then converted to a unified time scale according to the actual playback timestamps corresponding to the effective contribution frames, ultimately yielding a verification motion value. This verification motion value is then used in step S106 to form the device control value.

[0181] For example, continuing with the example of the contribution frame intervals allowed for output based on the above interval gating results, since frames 99, 100, and 101 all meet the scene admission conditions, all three video frames are determined as valid contribution frames. Frames 99 and 100 are respectively paired as the first valid frame pair, and frames 100 and 101 are paired as the second valid frame pair. For the first valid frame pair, using frame 99 as the reference video frame, assuming the average motion vector of the target region is (4,1) and the average motion vector of the background region is (1,0) based on dense optical flow, the target motion feature corresponding to this valid frame pair is (3,1). For the second valid frame pair, using frame 100 as the reference video frame, assuming the average motion vector of the target region is (5,1) and the average motion vector of the background region is (1,0), the corresponding target motion feature is (4,1). The cumulative target motion of the two valid frame pairs is (7,2). Based on the playback timestamp obtained in step S101, the inter-frame time of both valid frame pairs is approximately 0.0333 seconds; therefore, the effective time length is approximately 0.0667 seconds. When the device adopts a uniform control time length of approximately 0.0333 seconds, the cumulative target motion is normalized according to the effective time length, resulting in a verification motion value of approximately (3.5, 1). Therefore, step S104 does not directly use the cumulative motion of the two valid frame pairs as the basis for device control; instead, it normalizes the motion based on the actual effective time length involved in the motion analysis and provides the resulting verification motion value to step S106.

[0182] Step S105: When the interval gating result is prohibited from output, the verification motion value at the target control moment is set to a safe value.

[0183] The detailed implementation steps of step S105 include: Step S1051: Determine the safety value. Specifically, the safety value uses the same data format as the normal verification motion value obtained in step S104, and is used to indicate that no target motion drive is generated at the current target control moment.

[0184] When the verification motion value is represented by a two-dimensional motion vector, then: The value is determined to be the safe value.

[0185] in, This represents a safety value, and its vector dimension is the same as the verification motion value. The vectors have the same dimension.

[0186] When the controlled device uses a non-zero device control value to represent a no-action state, the device interface code is not directly used as a safety value. Instead, the aforementioned safety value is still used at the motion value data level and converted into the corresponding no-action control value of the controlled device in step S106 through the device control mapping relationship.

[0187] Thus, a safety value is obtained that is at the same data level as the normal verification motion value.

[0188] Step S1052: Set the verification motion value at the target control moment to a safe value. Specifically, when step S103 obtains: .

[0189] At that time, the motion analysis results corresponding to the current contribution frame interval are not read as input for device control curve generation, but instead the following is executed: Therefore, the verification motion value corresponding to the target control moment is set as a safety value.

[0190] In step S105, when the interval gating result is prohibited from output, a verification motion value of the same data type as the normal verification motion value in step S104 is still generated. Therefore, the verification motion values ​​formed in steps S104 and S105 can directly enter step S106 for device control value mapping without having to reread the original video frame for judgment in step S106.

[0191] For example, for another target control moment, suppose its motion characteristics also need to be formed by three consecutive video frames. The middle video frame has entered the target scene and the target scene confidence level reaches the scene admission threshold, while the previous video frame is still in the black screen end stage, and the next video frame has entered the shot switching stage. Furthermore, the target scene confidence levels corresponding to both the previous and next video frames are lower than the scene admission threshold. In this case, only the middle video frame in the contribution frame interval meets the scene admission condition. There are no two adjacent video frames in the video frame sequence that both meet the scene admission condition. Therefore, step S103 determines the interval gating result of the contribution frame interval as prohibiting output. Step S105 no longer calculates the normal target motion characteristics based on the contribution frame interval, but instead directly sets the verification motion value of the target control moment to a safe value; when the verification motion value is represented by a two-dimensional motion vector, it can be set to (0,0). The safety value is then entered into step S106 in the same data format as the normal verification motion value obtained in step S104, so that the image displacement caused by the black screen change and lens switching will not be used as the motion driving quantity of the current target control moment and enter the device control curve.

[0192] Step S106: Generate and output the device control curve based on the verification motion value corresponding to the continuous target control time and the playback timestamp. The detailed implementation steps of step S106 include: Step S1061: Map the verification motion value to the device control value. Specifically, before the video-driven device control starts, establish a mapping relationship between the verification motion value and the device control value.

[0193] In one implementation, when the controlled device forms a single device control intensity based on the target motion degree, the corresponding equivalent motion speed can be determined based on the verification motion value obtained in step S104, and the device control value can be determined based on the equivalent motion speed through a nonlinear response relationship.

[0194] The first The verification motion value corresponding to each target control moment is denoted as The preset control time length used in step S104 is denoted as The first value is determined based on the vector magnitude of the verified motion value. The equivalent motion velocity corresponding to each target control moment: .

[0195] in, Indicates the first The equivalent motion velocity corresponding to each target control moment; The vector magnitude representing the verified motion value; This indicates the preset control time length.

[0196] Set silent speed threshold Reference speed and curve index ,in, The nonlinear response value is determined based on the equivalent motion velocity: .

[0197] in: .

[0198] Wherein, the silent speed threshold Used to filter out small movements caused by image noise, optical flow calculation errors, or minute jitter of the target object; the reference velocity Used to determine the intermediate response position of a nonlinear response curve, when At that time, the nonlinear response value is 0.5; the curve exponent Used to adjust the degree of response growth corresponding to different speed ranges.

[0199] The original control strength corresponding to the current target control moment is obtained based on the nonlinear response value: .

[0200] in, Indicates the original control strength. This indicates the maximum control strength that the controlled device is allowed to output; when the controlled device uses a normalized strength range of 0 to 100, the... It can be set to 100.

[0201] To reduce the probability of sudden changes in the control strength of the equipment at consecutive target control moments, different time constants are used for smoothing based on the direction of change of the current original control strength relative to the control strength of the equipment at the previous target control moment.

[0202] The first The time constant corresponding to each target control moment is denoted as . ,but: .

[0203] in, This represents the first time constant used when the equipment control intensity increases. This represents the second time constant used when the equipment control intensity decreases. It is set when a rapid increase and slow decrease in equipment control intensity is required. .

[0204] The first The playback time position corresponding to each target control moment is denoted as . The playback time position corresponding to the previous target control moment is recorded as... Then the time interval between adjacent target control moments is: .

[0205] The smoothing coefficient is determined based on the time interval and time constant: .

[0206] And determine the first based on the smoothing coefficient. The equipment control intensity corresponding to each target control moment: .

[0207] in, This indicates the equipment control intensity corresponding to the previous target control moment. This indicates the equipment control intensity corresponding to the current target control moment.

[0208] For example, when the control intensity of the equipment increases, the first time constant can be... Setting it to 55ms allows the device control strength to quickly follow the original control strength; when the device control strength decreases, the second time constant can be adjusted. Setting it to 420ms allows the control intensity of the device to gradually decrease. The 55ms and 420ms mentioned above are just examples and can be adjusted according to the actual dynamic response characteristics of the controlled device.

[0209] The silent speed threshold The reference velocity can be determined based on historical velocity samples obtained when the target object is stationary; The curve index can be determined based on the motion speed corresponding to the predetermined intermediate control intensity during the equipment calibration process; The first time constant can be determined by parameter traversal or curve fitting based on the correspondence between multiple motion velocities and the desired control intensity; Second time constant It can be determined based on the rise response time and fall response time of the controlled equipment.

[0210] In another implementation, the verification motion values ​​and equipment control values ​​can also employ a piecewise linear mapping relationship based on equipment calibration points. This mapping relationship can be read from the control configuration file of the controlled equipment or formed during the equipment calibration process.

[0211] During the equipment calibration process, multiple calibration verification motion values ​​are given sequentially, and the equipment control values ​​corresponding to each calibration verification motion value are recorded, thereby forming multiple mapping calibration points.

[0212] The first The equipment control value corresponding to each target control moment is denoted as: ,but: .

[0213] in, This indicates the mapping relationship between the verification motion values ​​and the equipment control values.

[0214] When the controlled device forms a device control quantity based solely on the target motion degree, the vector magnitude of the verification motion value can be read as the input of the mapping relationship.

[0215] The current motion quantity used for mapping is denoted as... If two adjacent calibrated motion quantities are respectively and The corresponding equipment control values ​​are as follows: and And satisfy: .

[0216] as well as: .

[0217] The equipment control value is then obtained according to the following formula: .

[0218] in, This indicates the equipment control value corresponding to the current exercise level.

[0219] When the controlled device has multiple device control dimensions corresponding to different motion components of the verification motion value, the corresponding motion components in the verification motion value are read respectively, and the device control components are obtained by using the calibration mapping relationship corresponding to each device control dimension. The device control components together form the device control value at the current target control moment.

[0220] For the safety value generated in step S105, the device control value corresponding to the inactive state of the controlled device is obtained according to the same mapping relationship.

[0221] Therefore, the verification motion values ​​corresponding to the continuous target control time are converted into the corresponding equipment control values.

[0222] Step S1062: Generate a device control curve based on the playback timestamp, such as... Figure 8 As shown, the device control values ​​corresponding to multiple target control moments are arranged according to the video playback time axis to form a device control curve.

[0223] The first The control points corresponding to each target control moment are denoted as: .

[0224] in, Indicates the first One control point; This indicates the time position of the corresponding target control moment on the video playback timeline; This indicates the equipment control value corresponding to the target control moment.

[0225] according to: The control points are arranged in the following order, among which, This indicates the total number of target control moments.

[0226] When the controlled equipment requires continuously changing control values, for adjacent control points and Determine the position of any time between the two according to the following formula. Corresponding equipment control values: .

[0227] in: .

[0228] and: .

[0229] When the device control value consists of multiple control components, the above interpolation calculation is performed on each device control component separately.

[0230] The device control values ​​formed between adjacent control points are connected in sequence according to the playback time to obtain the device control curve.

[0231] It should be noted that the interpolation between adjacent control points in this step is used to determine the equipment control value between two target control times to form a time-continuous equipment control curve. This is independent of the mapping relationship used between the verification motion value and the equipment control value in step S1061, and is not used to limit the linear mapping relationship between the verification motion value and the equipment control value.

[0232] In another implementation, when the controlled device performs control according to discrete control points, no interpolation is performed between adjacent control points. Instead, the corresponding device control values ​​are output sequentially according to the time position corresponding to each target control time.

[0233] Therefore, the device control curve is obtained based on the verification motion value corresponding to the continuous target control moment and its playback time position in the video playback time axis.

[0234] Step S1063: Output the device control curve according to the playback timestamp. Specifically, during video playback, obtain the current video playback time and match the current video playback time with the time position of the device control curve.

[0235] When the current video playback time reaches the corresponding target control time or is located between the corresponding adjacent control points, the device control value corresponding to the device control curve is read and the device control value is output to the controlled device.

[0236] When the normal verification motion value is obtained from step S104 at the corresponding target control time, the device control value mapped from the normal verification motion value is output; when the safety value is obtained from step S105 at the corresponding target control time, the no-action control value mapped from the safety value is output.

[0237] In step S106, the verification motion values ​​formed in step S104 or step S105 for each target control moment are mapped to the control data level of the controlled device, and the device control curve is formed and output according to the video playback timeline.

[0238] For example, continuing with the verification motion value obtained in step S104, assuming the target control time is 3.300 seconds, the corresponding verification motion value is (3.5, 1). When the controlled device only forms a device control quantity based on the target motion degree, the vector magnitude of the verification motion value is calculated as the mapping input, and the device control value corresponding to the target control time is determined according to the pre-established calibration mapping relationship between the verification motion value and the device control value. Assuming the device control values ​​corresponding to the three consecutive target control times of 3.267 seconds, 3.300 seconds, and 3.333 seconds are denoted as follows: , and Then control points are formed respectively. , and The data is arranged in order of playback timestamp. When the controlled device requires continuously changing device control values, the device control value corresponding to the intermediate playback time is determined between adjacent control points using the interpolation method described in step S1062. The device control curve is formed by the control values ​​of multiple consecutive control points and adjacent control points. When a target control time is set to a safe value in step S105, a corresponding no-action control value is formed at that target control time through the same mapping relationship. When the video actually plays to the corresponding playback time position, the video processing device reads the corresponding device control value from the device control curve and outputs it to the controlled device, thereby completing the video motion analysis result to the actual device control output.

[0239] like Figure 9 As shown, this application also provides a video-driven device control system based on contribution frame interval verification, including a video acquisition module, a contribution interval determination module, an interval gating module, a motion verification module, and a curve output module.

[0240] The video acquisition module is used to acquire a video frame sequence and the playback timestamp corresponding to each video frame in the video frame sequence. The specific execution process corresponds to step S101.

[0241] The contribution interval determination module is used to determine the contribution frame interval participating in motion feature calculation from the video frame sequence for the target control time, and to obtain the target scene confidence of each video frame within the contribution frame interval. The specific execution process corresponds to step S102.

[0242] The interval gating module is used to determine the effective scene coverage ratio of the contributing frame interval based on the confidence level of each target scene, and to determine the interval gating result of the contributing frame interval in combination with the scene change status of the video frames at the interval boundary of the contributing frame interval. The specific execution process corresponds to step S103.

[0243] The motion verification module is used to filter valid contribution frames from the contribution frame interval when the interval gating result is allowed to output, determine the target motion features of the target area relative to the background area based on the valid contribution frames, and perform time normalization based on the playback timestamps corresponding to the valid contribution frames to obtain the verification motion value; it is also used to set the verification motion value to a safe value when the interval gating result is prohibited to output, and the specific execution process corresponds to steps S104 and S105 respectively.

[0244] The curve output module is used to generate and output the device control curve based on the verification motion value corresponding to the continuous target control time and the playback timestamp. The specific execution process corresponds to step S106.

[0245] The video acquisition module, contribution interval determination module, interval gating module, motion verification module, and curve output module can be implemented by the processor executing a computer program stored in the memory.

[0246] The above description is merely a preferred embodiment of this application. It should be noted that, due to the limitations of written expression, there are objectively infinite specific structures. For those skilled in the art, without departing from the principles of this application, several improvements, modifications, or variations can be made, and the above technical features can be combined in an appropriate manner. These improvements, modifications, variations, or combinations, or the direct application of the concept and technical solution of this application to other situations without modification, should all be considered within the scope of protection of this application.

Claims

1. A video driving device control method based on contribution frame interval check, characterized by, include: Obtain the video frame sequence and the playback timestamp corresponding to each video frame in the video frame sequence; For the target control moment, the contributing frame intervals involved in motion feature calculation are determined from the video frame sequence, and the target scene confidence of each video frame within the contributing frame interval is obtained; The effective scene coverage ratio of the contributing frame interval is determined based on the confidence level of each target scene, and the interval gating result of the contributing frame interval is determined by combining the scene change status of the video frames at the interval boundaries of the contributing frame interval. When the interval gating result is allowed to output, valid contribution frames that meet the scene admission conditions are selected from the contribution frame interval. The target motion features of the target area relative to the background area are determined according to the valid contribution frames. The target motion features are time normalized according to the playback timestamp corresponding to the valid contribution frames to obtain the verification motion value of the target control time. When the interval gating result is output prohibited, the verification motion value at the target control moment is set to a safe value; The device control curve is generated and output based on the verification motion value corresponding to the continuous target control time and the playback timestamp.

2. The video driving device control method based on contribution frame interval check according to claim 1, characterized in that, Determine the contributing frame interval for the target control time, and obtain the target scene confidence of each video frame within the contributing frame interval, including: Based on motion characteristics, the range of preceding and subsequent video frames called is calculated to determine the time support range corresponding to the target control moment; The video frames whose playback timestamps are within the time support range and actually participate in motion feature calculation are read from the video frame sequence and arranged according to the playback timestamps to form the contribution frame interval; Target scene determination is performed on each video frame within the contribution frame interval to obtain the target scene confidence level for each video frame.

3. The method according to claim 2, wherein, Determining the effective scene coverage ratio based on the confidence level of each target scene includes: The confidence scores of each target scene are compared with the scene admission threshold, and the number of video frames whose target scene confidence scores reach the scene admission threshold is counted. The ratio between the number of video frames and the total number of video frames within the contributing frame interval is determined as the effective scene coverage ratio.

4. The method of claim 3, wherein the method further comprises: The interval gating result is determined by combining the scene change state of the boundary video frames of the contribution frame interval, including: Based on the change in target scene confidence between the starting boundary video frame and its adjacent video frames within the contribution frame interval, and the change in target scene confidence between the ending boundary video frame and its adjacent video frames within the contribution frame interval, the scene change status of the two interval boundaries is determined respectively. When the effective scene coverage ratio reaches the coverage ratio threshold, the starting boundary video frame and the ending boundary video frame both meet the scene stability condition, and there is at least one set of adjacent video frames in the video frame sequence that both meet the scene admission condition within the contributing frame interval, the interval gating result is determined to be allowed output; otherwise, the interval gating result is determined to be prohibited output.

5. The method of claim 4, wherein the method further comprises: Determining the target motion features based on the effective contribution frames includes: Video frames whose target scene confidence reaches the scene admission threshold are determined as the effective contribution frames. Two adjacent video frames in the video frame sequence that are both effective contribution frames are formed into an effective frame pair. The video frame with the earlier playback timestamp in each effective frame pair is determined as the reference video frame. For each valid frame pair, a target region and a background region are determined from the reference video frame. The target region motion amount and the background region motion amount are calculated respectively. The difference between the target region motion amount and the background region motion amount is determined as the target motion feature of the corresponding valid frame pair.

6. The method of claim 5, wherein the method further comprises: Time normalization of the target motion features includes: The inter-frame time of each valid frame pair is determined based on the playback timestamps corresponding to the two valid contributing frames in each valid frame pair, and the inter-frame times are added together to obtain the effective time length. The target motion features corresponding to each effective frame are accumulated, and the accumulated target motion is converted to the motion scale corresponding to the preset control time length according to the effective time length. The converted target motion is determined as the verification motion value of the target control time.

7. The method of claim 1, wherein the method further comprises: Setting the verification motion value to a safe value includes: Based on the data format of the verification motion value, a safety value is determined that indicates that no target motion drive is generated at the target control moment and has the same data format as the verification motion value; When the interval gating result is prohibited from output, the motion analysis result corresponding to the contribution frame interval is not used as the basis for generating the device control curve, and the verification motion value corresponding to the target control time is set as the safety value.

8. The method of claim 1, wherein the method further comprises: Generate and output the device control curve, including: Based on the pre-established mapping relationship between the verification motion values ​​and the equipment control values, the verification motion values ​​corresponding to each target control moment are converted into equipment control values; The playback timestamp corresponding to each target control moment is combined with the corresponding device control value to form multiple control points, and the multiple control points are arranged according to the playback timestamp to form the device control curve; During video playback, the corresponding device control value in the device control curve is read and output according to the current playback time.

9. A video driving device control method based on contribution frame interval verification according to claim 4, characterized in that, The determination of the coverage ratio threshold and the boundary change threshold used to form the scene stability condition includes: Obtain the first sample contribution frame interval pre-labeled as allowed output and the second sample contribution frame interval pre-labeled as prohibited output. Set multiple candidate coverage ratio thresholds according to the effective scene coverage ratio of each sample contribution frame interval. Count the number of interval misjudgments corresponding to each candidate coverage ratio threshold. Determine the candidate coverage ratio threshold with the smallest number of interval misjudgments as the coverage ratio threshold. Obtain first boundary samples pre-labeled as stable scene states and second boundary samples pre-labeled as changing scene states. Set multiple candidate boundary change thresholds based on the target scene confidence difference between adjacent video frames in each boundary sample. Count the number of boundary state misjudgments corresponding to each candidate boundary change threshold and determine the candidate boundary change threshold with the smallest number of boundary state misjudgments as the boundary change threshold.

10. A video driving device control system based on contribution frame interval verification, implementing the video driving device control method based on contribution frame interval verification as described in any one of claims 1 to 9, characterized in that, include: The video acquisition module is used to acquire a video frame sequence and the playback timestamp corresponding to each video frame in the video frame sequence; The contribution interval determination module is used to determine the contribution frame intervals involved in motion feature calculation from the video frame sequence for the target control time, and to obtain the target scene confidence of each video frame within the contribution frame interval. The interval gating module is used to determine the effective scene coverage ratio of the contributing frame interval based on the confidence level of each target scene, and to determine the interval gating result of the contributing frame interval by combining the scene change status of the video frames at the interval boundaries of the contributing frame interval. The motion verification module is used to filter valid contribution frames that meet the scene admission conditions from the contribution frame interval when the interval gating result is allowed output, determine the target motion features of the target area relative to the background area based on the valid contribution frames, and perform time normalization on the target motion features based on the playback timestamp corresponding to the valid contribution frames to obtain the verification motion value at the target control time. And when the interval gating result is prohibited from output, the verification motion value at the target control moment is set to a safe value; The curve output module is used to generate and output the device control curve based on the verification motion value corresponding to the continuous target control time and the playback timestamp.

Images