A target recognition processing method based on a sliding window and multi-view fusion
By using multi-view fusion and sliding window processing, the problems of non-steady-state fluctuations and occlusion in single-view visual recognition are solved, achieving high-precision target recognition and improving the stability and accuracy of recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DMAI (GUANGZHOU) CO LTD
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-09
Smart Images
Figure CN122176607A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visual recognition technology, and more specifically to a target recognition processing method based on sliding window and multi-view fusion. Background Technology
[0002] In the field of visual recognition technology, although single-view target detection and behavior recognition technologies have made significant progress in scenarios such as security monitoring, autonomous driving, and industrial manufacturing, they still face two key technical bottlenecks: one is the non-steady-state fluctuation problem of single-view temporal dimension recognition results; the other is the problem of missing key information caused by visual occlusion.
[0003] First, in single-view recognition scenarios, deep neural network models are typically used to perform frame-by-frame analysis of video data, which can lead to significant fluctuations in recognition results between adjacent frames. Existing simple filtering methods based on confidence thresholds can only filter out low-confidence detection results, failing to effectively address the inconsistency of the model in the temporal dimension. Furthermore, due to the lack of an adaptive correction mechanism based on temporal correlation, once a misidentification occurs, this error accumulates in subsequent frames, causing a sharp decline in system performance. Second, the inherent blind spots of a single viewpoint mean that some key actions or details may not be captured by the camera. The lack of an effective multi-source information fusion mechanism makes it impossible to reliably infer occluded areas, leading to frequent missed recognitions. Especially in practical vocational skills assessment scenarios where recognition accuracy is extremely important, these misidentifications and missed recognitions pose a significant challenge to the system's usability. Summary of the Invention
[0004] In view of the above problems, the present invention proposes a target recognition processing method and system based on sliding window and multi-view fusion, so as to overcome the above problems or at least partially solve the above problems.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A target recognition processing method based on sliding window and multi-view fusion includes the following steps: Acquire video streams from multiple perspectives of the target scene; For each video segment from each perspective, visual features of the video frames are extracted, and targets in the video frames are preliminarily classified and identified to obtain basic recognition results under a single perspective. Determine whether the basic recognition results of multiple consecutive frames under each single view conform to the preset business rules. If yes, mark the result as 1; otherwise, mark the result as 0 and generate a single view result sequence. Frame-level fusion of visual features from each single viewpoint is performed to obtain fused visual features from multiple frames across multiple viewpoints. Determine whether the fused visual features of each frame conform to the preset spatial rules. If yes, mark the result as 1; otherwise, mark the result as 0 and generate a multi-view result sequence. The single-view result sequence and the multi-view result sequence are smoothed and fused respectively to generate the recognition result of the current video segment.
[0007] Furthermore, at least six cameras are used to capture video streams covering the operator's operation process in the target scene; the six cameras are: a dynamic head-mounted camera, and a static head-up camera, panoramic camera, left-view camera, right-view camera and top-view camera; Head-mounted cameras are used to record actions and operational details within the operator's field of vision in real time; The head-up camera is positioned directly in front of the workstation, covering the operator's entire body when standing. The panoramic camera is positioned directly in front of the workstation and is higher than the usual camera, covering the operator's overall working environment. The left-view camera is positioned on the left side of the workstation to capture the operator's left-side actions. The right-view camera is positioned on the right side of the workstation to capture the operator's actions on the right side. The overhead camera is positioned above the workstation to capture the operator's entire operating process.
[0008] Furthermore, a pre-trained YOLO model is used as the base model. For each video stream from a different viewpoint, the base model is used to perform preliminary classification and recognition of targets in the video frames, obtaining each target and its location in the video frame, which serves as the basic recognition result for a single frame from a single viewpoint. The business rule execution logic for each detection point that meets the target scenario is encapsulated into three business components, including spatial cognitive reasoning component, temporal cognitive reasoning component and spatiotemporal cognitive reasoning component; The three business components determine whether the basic recognition results of each single viewpoint across multiple frames conform to the preset business rules from both temporal and spatial dimensions, and each business component outputs a single viewpoint result sequence.
[0009] Furthermore, the business rule execution logic of the spatial cognitive reasoning component is as follows: For any two targets to be detected in each video frame, set a position threshold for the spatial relationship between the two targets. and depth threshold ; Calculate the intersection-union ratio (IUU) of the two target bounding boxes; Calculate the depth value between two targets. The depth value represents the relative front and back positions of the two targets and reflects the distance between the targets and the camera. To determine whether two targets satisfy a spatial relationship, the method is: when the Intersection over Union (IoU) of the bounding boxes of the two targets is greater than a position threshold. Furthermore, the depth values of both targets are greater than the depth threshold. If the business rules are met, the result is marked as 1; otherwise, the result is marked as 0. Record the spatial relationship judgment results for multiple consecutive frames as a single-view result sequence of the spatial cognitive reasoning component.
[0010] Furthermore, the business rule execution logic of the time-series cognitive reasoning component is as follows: Define the sequence of recognition results for a target in a video clip under two states, i and j. and ;in, The sequence records the recognition results for target state i. In the sequence, 1 indicates that state i has been identified, and 0 indicates that state i has not been identified; The sequence records the recognition results for target state j. In the sequence, 1 indicates that state j has been identified, and 0 indicates that state j has not been identified; Starting from time t, the identification results of the two states of the target will be appended to the sequence respectively. and inside; At time t+n, for the sequence and sequence Perform an XOR operation, then convert the data to decimal for comparison to obtain the final result. If the corresponding values of the two sequences at the same time are different, the XOR result is 1. If the corresponding values of the two sequences are different at every time, the XOR result is a sequence of all 1s, which is converted to decimal as follows: ; If two sequences and The XOR result is equal to If the result is 1, it means that the target has completed the complete temporal change from state i to state j from time t to time t+n. In this case, the single-view judgment result of the target is recorded as 1. Otherwise, it means that the state transition is incomplete and is recorded as 0. Record the results of the target's state transition judgment in other time periods of the video clip; The results of state transition judgments for all time periods in the video clip are summarized and used as a single-view result sequence for the temporal cognitive reasoning component.
[0011] Furthermore, the business rule execution logic of the spatiotemporal cognitive reasoning component is as follows: Set the distance threshold for detecting changes in the target's position. and angle threshold ; Identify video clips Time and The coordinates of the target to be detected within the bounding box in the field of view at all times; Calculate the Euclidean distance between two coordinate points ; Calculate the deviation angle between two coordinate points ; if ,and This indicates that the target to be detected came from Time's up If the correct position change occurs at any time, the single-view judgment result of the target to be detected is recorded as 1; otherwise, it is recorded as 0. Record the results of positional changes of the target to be detected in other time periods within the video clip; The results of positional changes across all time periods in the video clip are summarized and used as a single-view result sequence for the spatiotemporal cognitive reasoning component.
[0012] Furthermore, the YOLO model is improved to obtain a fusion feature recognition model. The improvement involves adjusting the input layer of the YOLO model to receive multiple single-view visual features, and adding a cross-view feature fusion module between the backbone and neck to fuse the multiple single-view visual features. Based on the fusion feature recognition model, frame-level fusion and spatial rule judgment are performed on the single-view single-frame visual features of each path. The specific process includes: Visual features of the first video frame for each viewpoint Calculate the query vector, key vector, and value vector, where n represents the total number of viewpoints. For each business detection point, determine the attention weights between the primary viewpoint i and other viewpoints j. :
[0013] in, K represents the key vector of the j-th viewpoint. j The transpose of the matrix; The query vector representing the primary viewpoint i; Attention weights Normalization is performed to obtain normalized weights. ; The fused visual features are calculated based on the normalized weights:
[0014] in, Indicates the fusion of visual features. The visual features representing the primary visual field i Indicates the adjustment factor; Represents a value vector from other perspectives; Spatial analysis is performed on the fused visual features. If the target position, depth, and occlusion relationship all conform to the preset spatial rules, it is recorded as 1; otherwise, it is recorded as 0. The spatial analysis results of other video frames under each visual path are recorded and summarized as a multi-view result sequence of the fusion feature recognition model.
[0015] Furthermore, before performing frame-level fusion of the visual features from each single viewpoint and single frame, the following steps are also included: Align the time of each camera with the time of the NTP server; During video capture, OSD timestamps are added to each video stream and set to a uniform format; At the same frequency, frames are extracted from each video stream to obtain images to be processed. The OSD timestamps on the images are identified to extract time information. Based on the extracted time information, the images are classified and sorted in chronological order, and the group of images with the closest timestamps is selected as a set of images from different perspectives at the same time. Feature extraction is performed on the selected image set to obtain the visual features from various perspectives. .
[0016] Furthermore, smoothing is performed on each single-view result sequence and multi-view result sequence. The processing includes: Dynamically select the time span based on the business scenario and define a sliding window; Based on the sliding window size, record the corresponding length of the discretized recognition result sequence, represented as follows: ,in ; For each result within the sliding window, the mode is counted. If the mode is unique within the window, the smoothed result is equal to that mode. If there are multiple modes within the window, the mode that appears most frequently is taken as the smoothed result.
[0017] Furthermore, the method for determining the recognition result of the current video segment is as follows: After smoothing each result sequence, the smoothed result is output. The average or mode of each smoothed result is calculated as the final identification result.
[0018] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects: 1. This invention effectively overcomes the blind spots and viewpoint limitations in single-view recognition by integrating multi-view feature fusion. The integration of feature information from different perspectives enhances the richness and accuracy of the features, thereby improving the precision and robustness of the recognition. By introducing multi-view feature fusion to integrate feature information from different perspectives and constructing multi-scale feature representations, this fusion method can fully utilize multi-view data and improve the model's adaptability to complex scenes.
[0019] 2. This invention can determine the relationship between targets from both temporal and spatial dimensions, enhancing the model's reasoning ability. This reasoning method can effectively identify changes in the state and position of targets, improving the model's ability to understand and recognize complex scenes, thereby enhancing the accuracy and reliability of identification.
[0020] 3. This invention reduces the impact of perspective changes and noise interference by smoothing video frames, thereby improving the stability and accuracy of the results. By integrating multiple single-view recognition results and multi-view fusion recognition results, the accuracy and reliability of the recognition results are further improved, effectively reducing false recognition and missed recognition. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0022] Figure 1 This is a flowchart of the target recognition processing method based on sliding window and multi-view fusion provided in the embodiments of the present invention; Figure 2 This is a framework diagram of the target recognition processing method based on sliding window and multi-view fusion provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the arrangement of multi-view cameras provided in an embodiment of the present invention. Detailed Implementation
[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0024] like Figures 1-2As shown, this embodiment of the invention discloses a target recognition processing method based on sliding window and multi-view fusion, including the following steps: S1. Acquire video streams from multiple perspectives of the target scene; S2. For each video segment from each viewpoint, extract the visual features of the video frames and perform preliminary classification and recognition of the targets in the video frames to obtain the basic recognition results under a single viewpoint. S3. Determine whether the basic recognition results of multiple consecutive frames under each single view conform to the preset business rules. If yes, mark the result as 1; otherwise, mark the result as 0 and generate a single view result sequence. S4. Perform frame-level fusion of visual features from each single viewpoint to obtain fused visual features from multiple frames under multiple viewpoints. Determine whether the fused visual features of each frame conform to the preset spatial rules. If yes, mark the result as 1; otherwise, mark the result as 0 and generate a multi-view result sequence. S5. Smooth and fuse the single-view result sequences and multi-view result sequences respectively to generate the recognition result of the current video segment.
[0025] The following provides further explanation of each of the above steps.
[0026] In S1, taking practical vocational skills training and evaluation as an example, in order to capture the operation data completed by the operator, at least six perspectives are required: a moving head-mounted camera and static head-up cameras, panoramic cameras, left-view cameras, right-view cameras, and top-view cameras. The arrangement is as follows: Figure 3 As shown, these six cameras comprehensively capture the operator's actions, including inspecting tools and using them to perform actual testing on instruments. This provides high-quality data support for subsequent target detection and behavior recognition.
[0027] The head-mounted camera is an action camera worn by the operator to capture the operation process from a first-person perspective. This camera can record the operator's movements and operational details within their field of vision in real time, including hand gestures and tool usage.
[0028] The head-up camera is positioned directly in front of the workstation, covering the operator's entire body when standing; this camera can record the operator's actions in different positions, providing a global perspective.
[0029] The panoramic camera is positioned directly in front of the operator's workstation and is higher than the usual camera, covering the operator's overall operating environment; this camera can record the operator's actions in different positions and provide a global perspective.
[0030] The left-view camera is positioned on the left side of the workstation to capture the operator's actions on the left side, ensuring that the operator's tool usage and operational details on the left side can be recorded.
[0031] The right-view camera is positioned on the right side of the workstation to capture the operator's actions on the right side, ensuring that the operator's tool usage and operational details on the right side can be recorded.
[0032] The overhead camera is positioned above the workstation to capture the operator's entire operation process, ensuring that the operator's overall actions and tool usage are recorded.
[0033] S2. This invention uses YOLOv10 or YOLOv10-Pose as the core basic model. The input video stream is sampled at a frequency of 1 frame per second or 2 frames per second. The sampled images are then used with the basic model set to obtain the expected target or human pose. The key work at this stage is multi-view site setup and data acquisition; subsequent data annotation, model training, and deployment are not detailed here.
[0034] Once the basic model is trained, it can detect targets and their locations from input images or videos, including key points of human posture (such as hands, head, feet, etc.), as the basic recognition results for a single frame from a single viewpoint.
[0035] S3. In practical detection scenarios, simply detecting the target is insufficient; it's also necessary to determine the relationships between targets from both temporal and spatial dimensions. The execution logic of detection point business rules that meet specific scenarios is encapsulated into reusable business components, including spatial cognitive reasoning, temporal cognitive reasoning, and spatiotemporal cognitive reasoning components. These three business components determine whether the basic recognition results across multiple frames from each single perspective conform to preset business rules from both temporal and spatial dimensions, and each component outputs a single-view result sequence.
[0036] The specific execution process of the three business components will be explained in detail below.
[0037] 1) Multiple targets in an image, such as a safety helmet and key points of the human body (hands, feet, head, etc.), have positional relationships that constitute a spatial grammatical structure. These positional relationships also convey specific semantic information. Therefore, the business rule execution logic of the spatial cognitive reasoning component is as follows: ① For any two targets to be detected in each video frame, set a position threshold for the spatial relationship between the two targets. and depth threshold ; ② Calculate the intersection-union ratio (IU / U) of the two target bounding boxes; assume the coordinates of the two bounding boxes A and B are respectively ( , , , ), and ( , , , ), then the IoU is calculated according to the following formula: IoU(A, B) =
[0038]
[0039]
[0040] ③ Calculate the depth value between two targets. The depth value represents the relative front - back position of the two targets, reflecting the distance relationship between the targets and the camera. The larger the depth value, the farther the target is, and the smaller the depth value, the closer the target is. By comparing the depth values, the occlusion relationship can be judged. If the depth value of A < B, then A is in front of B and may occlude B.
[0041] Depth =
[0042] Among them, is the depth value predicted by the model, is the maximum depth value in the depth map, which is used for normalization. The depth map has the same size as the original image, and the pixel value is the target depth value, presenting the relative distance between the target and the camera. The acquisition method is that the modified YOLOv10 model makes pixel - level depth prediction on multi - perspective frames to generate and normalize the depth map.
[0043] ④ Judge whether two targets meet the spatial relationship. The judgment method is: when the intersection - over - union IoU of the bounding boxes of the two targets is greater than the position threshold , and the depth values of the two targets are greater than the depth threshold , it means that the business rules are met, and the marking result is 1. Otherwise, the marking result is 0; Judge whether the value of the result sequence is greater than or equal to the preset window threshold (S >= Win). If the condition is met, output the result sequence for subsequent smoothing processing; if the condition is not met (N), then return to continue the result accumulation. Record the spatial relationship judgment results for multiple consecutive frames as the single - perspective result sequence of the spatial cognition reasoning component.
[0044] 2) In the practical vocational skill assessment business scenario, another detection point is to identify the state change of the target object at different time points (for example, the meter box door changes from the closed state to the open state). This recognition of the target state change over time is essentially an accurate expression of the temporal relationship. The business rule execution logic of the temporal cognition reasoning component is: ① Define the recognition result sequences and ;in, The sequence records the recognition results for target state i. In the sequence, 1 indicates that state i has been identified, and 0 indicates that state i has not been identified; The sequence records the recognition results for target state j. In the sequence, 1 indicates that state j has been identified, and 0 indicates that state j has not been identified; ② Starting from time t, the identification results of the two states of the target will be appended to the sequence respectively. and inside; ③ At time t+n, for the sequence and sequence Perform an XOR operation, then convert the data to decimal for comparison to obtain the final result. If the corresponding values of the two sequences at the same time are different, the XOR result is 1. If the corresponding values of the two sequences are different at every time, the XOR result is a sequence of all 1s, which is converted to decimal as follows: ; If two sequences and The XOR result is equal to If the result is 1, it means that the target has completed the complete temporal change from state i to state j from time t to time t+n. In this case, the single-view judgment result of the target is recorded as 1. Otherwise, it means that the state transition is incomplete and is recorded as 0.
[0045] ④ Record the results of the target's state transition judgment in other time periods of the video clip according to methods ②~③; ⑤ Determine whether the values of the result sequence are greater than or equal to the preset window threshold (S>=Win). If the condition is met, output the result sequence and perform subsequent smoothing processing; if the condition is not met (N), return to continue accumulating the results. Summarize the state transition judgment results of all time periods in the video clip as a single-view result sequence of the temporal cognitive reasoning component.
[0046] 3) The final detection point is identifying changes in the target's position, such as confirming whether an operator is climbing or descending a pole. This involves a combination of temporal and spatial reasoning. The business rule execution logic of the spatiotemporal cognitive reasoning component is as follows: ① Set the distance threshold for changes in the position of the target to be detected. and angle threshold ; ② Identify video clips Time and The coordinates of the target to be detected within the bounding box in the field of view at any given time. , )and( , ); ③ Calculate the Euclidean distance between two coordinate points. :
[0047] ④ Calculate the deviation angle between the two coordinate points. :
[0048] ⑤ Judge the result according to the following judgment formula:
[0049] if ,and This indicates that the target to be detected came from Time's up If the correct position change occurs at any time, the single-view judgment result of the target to be detected is recorded as 1; otherwise, it is recorded as 0. ⑥ Record the results of positional changes of the target to be detected in other time periods within the video clip, following the methods in ②~⑤; ⑦ Determine whether the values of the result sequence are greater than or equal to the preset window threshold (S>=Win). If the condition is met, output the result sequence and perform subsequent smoothing processing; if the condition is not met (N), return to continue accumulating the results. Summarize the position change judgment results of all time periods in the video clip as a single-view result sequence of the spatiotemporal cognitive reasoning component.
[0050] S4. The above steps demonstrate good recognition and reasoning capabilities for a single perspective. However, in the complex scenarios of vocational skills assessment, even with six cameras deployed to collect operational data, it is still difficult to completely avoid the problem of missing recognition information caused by blind spots and limited field of view. Furthermore, the differences in the shooting angles of each camera may also lead to misidentification. To overcome these technical challenges, this step introduces spatial analysis technology, aiming to further improve the accuracy and reliability of recognition by implementing a multi-view feature fusion scheme.
[0051] The first step in implementing multi-view feature fusion is to construct a multi-view, multi-scale fusion feature recognition model. Subsequently, in practical applications, the multi-view fusion features extracted by this model are used for recognition and inference. The logic for recognition and inference can draw upon and reuse the mature algorithm from step S2. Therefore, the core task of this step is to collect multi-view feature data and to train and optimize this fusion model.
[0052] The YOLOv10 network architecture is only suitable for processing a single image. To meet the recognition requirements of multi-view input scenes, it needs to be modified to obtain a fusion feature recognition model. The improvement method is to adjust the input layer of the YOLO model to receive multiple single-view visual features, add a cross-view feature fusion module between the backbone and neck, and use cross attention to allow the features of each view to interact with other views, extract key information, and then fuse the multiple single-view visual features.
[0053] Specifically, based on the fusion feature recognition model, frame-level fusion and spatial rule determination are performed on the visual features of each single viewpoint and single frame. The specific process includes: 1) In the spatial analysis phase, the first step is to perform temporal alignment processing on videos from multiple perspectives. This temporal alignment algorithm ensures synchronization of videos from different viewpoints over time, obtaining images from different perspectives at the same moment. The specific alignment method is as follows: Device time alignment: First, the time of each camera is aligned with the unified server time through the NTP server to ensure that each camera is capturing video data of the same action from different angles at the same time.
[0054] Timestamp calibration: During video capture, OSD timestamps (OnscreenDisplay) are added to each video stream and set to a uniform format (font, color, font size, background, etc.).
[0055] Timing Alignment: Frames are extracted from each video stream at the same frequency to obtain images to be processed. Then, the OSD time on the images is identified to extract the time information. Based on the identified time information, the images are classified and sorted according to time sequence. The group of images with the closest timestamps (usually within the allowable time error range) is selected as the image set from different perspectives at the same time.
[0056] 2) Based on the aforementioned basic model, feature extraction is performed on the selected image sets from each perspective to obtain the visual features of each perspective. Alternatively, you can directly input the fusion feature recognition model; or directly use the fusion feature model to extract features from a collection of images from multiple perspectives in parallel.
[0057] 3) Visual features of the first video frame for each viewpoint Calculate the query vector, key vector, and value vector, where n represents the total number of viewpoints; in this embodiment, n=6. The query vector, key vector, and value vector are calculated as follows:
[0058]
[0059]
[0060] Among them, Q i For the query vector, K i V is the key vector. i V is a value vector used to calculate attention weights; d is the dimension of the attention head; c is the number of channels in the feature map, i.e., V i The feature dimension is the number of features that represent a single-view feature.
[0061] 4) For each business detection point, determine the attention weights between the main viewpoint i and other viewpoints j. :
[0062] in, K represents the key vector of the j-th viewpoint. j The transpose of the matrix; The query vector representing the primary viewpoint i; The primary perspective is determined based on the core business testing needs, with the following one-to-one correspondence: Practical skills assessment (e.g., form box operation) → perspective directly facing the operation area (clear core testing target); Other business (e.g., target tracking) → primary perspective covering the entire target area (unobstructed, complete field of view). The core principle is: the primary perspective must maximize the capture of the features of the core business testing target.
[0063] 5) Attention weights Normalization is performed to obtain normalized weights. :
[0064] 6) Calculate the fused visual features based on the normalized weights:
[0065] in, Indicates the fusion of visual features. The visual features representing the primary visual field i Indicates the adjustment factor; Represents a value vector from other perspectives; 7) Perform spatial analysis on the fused visual features. If the target position, depth, and occlusion relationship (such as the relative front and back positions between targets) all conform to the preset spatial rules, then record it as 1; otherwise, record it as 0. 8) Record and summarize the spatial analysis results of other video frames under each vision path as a multi-view result sequence of the fusion feature recognition model.
[0066] S5. Each business component outputs a discrete sequence of recognition results, represented as follows: ,in Due to video noise, target occlusion, or model misjudgment, the original result sequence may exhibit two scenarios: High-frequency jitter: The recognition results of adjacent frames change frequently, such as the sequence segment: 1,0,1,0,1,0; Isolated noise: Transient misidentification leading to abnormal pulses, such as sequence fragments: 0,0,0,1,0,0,0.
[0067] Using the original sequence directly can lead to misjudgments in downstream tasks; therefore, temporal smoothing techniques are needed to improve the robustness of the results. This step designs a smoothing algorithm based on sliding window and mode statistics. The key steps are as follows: 1) Dynamically select the time span based on the business scenario and define the sliding window; typically, a detection action lasts for 5 seconds and is synchronized with the frame sampling frequency, for example, 2 frames are sampled per second, and W=10 corresponds to a 5-second time window. To ensure that each frame is covered, the sliding step size is designed to be 1 frame.
[0068] 2) Perform mode statistics on each result within the sliding window. If the mode is unique within the window, the smoothed result is equal to that mode; if there are multiple modes within the window, the mode that appears most frequently is taken as the smoothed result.
[0069] 3) Boundary processing The first W-1 frames use an incremental window, meaning that statistics are collected frame by frame from 1 to W. If there are fewer than W frames at the end of the sequence, the actual number of remaining frames is used for counting. The boundary handling method for the first W-1 frames is as follows: instead of using a fixed-size window W, the window size increases frame by frame starting from frame 1 (frame 1 window size = 1, frame 2 = 2, ..., frame W-1 = W-1). Each frame only counts the data of the currently existing frame, avoiding smoothing errors caused by insufficient window size (less than W frames) in the first W-1 frames.
[0070] The smoothing bias of the first W-1 frames is avoided by using an incremental window. The specific removal process is as follows: ① High-frequency jitter (such as a 1,0,1,0 sequence): The mode of the window is the value that appears most frequently (if there are 5 1s and 5 0s in the window, the result of the previous frame is used; if a certain value has a higher proportion, that value is output), eliminating frequent jumps between adjacent frames; ② Isolated noise (such as a single 1 mixed with multiple 0s): The mode of the window is 0, and the result after smoothing is 0, filtering out abnormal pulses that are misidentified momentarily.
[0071] After smoothing each result sequence, the average or mode of each smoothed result can be calculated as the final identification result.
[0072] The key to the entire process of this invention lies in: In the target recognition stage, the YOLOv10 target recognition model is used as the foundation. This model has powerful feature extraction and classification capabilities. The system acquires independent multi-view videos through video input and uses the YOLOv10 model to perform target detection on the images extracted from each video frame. It can accurately identify various targets and objects (such as tools, equipment, human bodies, etc.) in practical activities and output the corresponding detection results.
[0073] In the cognitive reasoning stage, a business component is designed for each business identification point to perform deep reasoning and logical judgment on the results of the target identification stage. The component performs logical reasoning on the temporal or spatial relationships of the result objects generated in the target identification stage. In terms of temporal relationships, the component judges whether the behavior of the target objects conforms to the preset time sequence logic; in terms of spatial relationships, the component analyzes whether the position, distance, and interaction patterns between target objects meet expectations.
[0074] In the spatial analysis phase, the videos from multiple perspectives are first subjected to temporal alignment. This alignment algorithm ensures that the videos from different perspectives are synchronized over time, capturing images from different viewpoints at the same moment. Subsequently, feature fusion is performed on these images, integrating feature information from different perspectives to enhance feature richness and accuracy. Based on this, the fused features are further segmented into feature information at different scales to construct multi-scale feature representations. These multi-scale feature information are used to train the model, enabling it to learn the feature representation of the target at different scales.
[0075] In the result smoothing stage, the result sequence obtained from cognitive reasoning and spatial analysis is optimized and smoothed using a sliding window, and a score fusion model is designed. First, based on the processing results of each previous stage, the confidence and score of target and behavior recognition are calculated to obtain a result sequence. Then, a sliding window is used to smooth the result sequence to eliminate noise and jitter, making the final result more stable and accurate.
[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0077] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A target recognition processing method based on sliding window and multi-view fusion, characterized in that, Includes the following steps: Acquire video streams from multiple perspectives of the target scene; For each video segment from each perspective, visual features of the video frames are extracted, and targets in the video frames are preliminarily classified and identified to obtain basic recognition results under a single perspective. Determine whether the basic recognition results of multiple consecutive frames under each single view conform to the preset business rules. If yes, mark the result as 1; otherwise, mark the result as 0 and generate a single view result sequence. Frame-level fusion of visual features from each single viewpoint is performed to obtain fused visual features from multiple frames across multiple viewpoints. Determine whether the fused visual features of each frame conform to the preset spatial rules. If yes, mark the result as 1; otherwise, mark the result as 0 and generate a multi-view result sequence. The single-view result sequence and the multi-view result sequence are smoothed and fused respectively to generate the recognition result of the current video segment.
2. The target recognition processing method based on sliding window and multi-view fusion as described in claim 1, characterized in that, At least six cameras should be used to capture video streams covering the operator's actions in the target scenario; The six cameras are: a dynamic head-mounted camera, and a static head-up camera, panoramic camera, left-view camera, right-view camera, and top-view camera; Head-mounted cameras are used to record actions and operational details within the operator's field of vision in real time; The head-up camera is positioned directly in front of the workstation, covering the operator's entire body when standing. The panoramic camera is positioned directly in front of the workstation and is higher than the usual camera, covering the operator's overall working environment. The left-view camera is positioned on the left side of the workstation to capture the operator's left-side actions. The right-view camera is positioned on the right side of the workstation to capture the operator's actions on the right side. The overhead camera is positioned above the workstation to capture the operator's entire operating process.
3. The target recognition processing method based on sliding window and multi-view fusion as described in claim 1, characterized in that, Using a pre-trained YOLO model as the base model, the base model is used to perform preliminary classification and recognition of targets in video frames for each viewpoint, obtaining the targets and their locations in the video frames, which serve as the basic recognition results for a single frame under a single viewpoint. The business rule execution logic for each detection point that meets the target scenario is encapsulated into three business components, including spatial cognitive reasoning component, temporal cognitive reasoning component and spatiotemporal cognitive reasoning component; The three business components determine whether the basic recognition results of each single viewpoint across multiple frames conform to the preset business rules from both temporal and spatial dimensions, and each business component outputs a single viewpoint result sequence.
4. The target recognition processing method based on sliding window and multi-view fusion as described in claim 3, characterized in that, The business rule execution logic of the spatial cognitive reasoning component is as follows: For any two targets to be detected in each video frame, set a position threshold for the spatial relationship between the two targets. and depth threshold ; Calculate the intersection-union ratio (IUU) of the two target bounding boxes; Calculate the depth value between two targets. The depth value represents the relative front and back positions of the two targets and reflects the distance between the targets and the camera. To determine whether two targets satisfy a spatial relationship, the method is: when the Intersection over Union (IoU) of the bounding boxes of the two targets is greater than a position threshold. Furthermore, the depth values of both targets are greater than the depth threshold. If the business rules are met, the result is marked as 1; otherwise, the result is marked as 0. Record the spatial relationship judgment results for multiple consecutive frames as a single-view result sequence of the spatial cognitive reasoning component.
5. The target recognition processing method based on sliding window and multi-view fusion as described in claim 3, characterized in that, The business rule execution logic of the time-series cognitive reasoning component is as follows: Define the sequence of recognition results for a target in a video clip under two states, i and j. and ;in, The sequence records the recognition results for target state i. In the sequence, 1 indicates that state i has been identified, and 0 indicates that state i has not been identified; The sequence records the recognition results for target state j. In the sequence, 1 indicates that state j has been identified, and 0 indicates that state j has not been identified; Starting from time t, the identification results of the two states of the target will be appended to the sequence respectively. and inside; At time t+n, for the sequence and sequence Perform an XOR operation, then convert the data to decimal for comparison to obtain the final result. If the corresponding values of the two sequences at the same time are different, the XOR result is 1. If the corresponding values of the two sequences are different at every time, the XOR result is a sequence of all 1s, which is converted to decimal as follows: ; If two sequences and The XOR result is equal to If the result is 1, it means that the target has completed the complete temporal change from state i to state j from time t to time t+n. In this case, the single-view judgment result of the target is recorded as 1. Otherwise, it means that the state transition is incomplete and is recorded as 0. Record the results of the target's state transition judgment in other time periods of the video clip; The results of state transition judgments for all time periods in the video clip are summarized and used as a single-view result sequence for the temporal cognitive reasoning component.
6. The target recognition processing method based on sliding window and multi-view fusion as described in claim 3, characterized in that, The business rule execution logic of the spatiotemporal cognitive reasoning component is as follows: Set the distance threshold for detecting changes in the target's position. and angle threshold ; Identify video clips Time and The coordinates of the target to be detected within the bounding box in the field of view at all times; Calculate the Euclidean distance between two coordinate points ; Calculate the deviation angle between two coordinate points ; if ,and This indicates that the target to be detected came from Time's up If the correct position change occurs at any time, the single-view judgment result of the target to be detected is recorded as 1; otherwise, it is recorded as 0. Record the results of positional changes of the target to be detected in other time periods within the video clip; The results of positional changes across all time periods in the video clip are summarized and used as a single-view result sequence for the spatiotemporal cognitive reasoning component.
7. The target recognition processing method based on sliding window and multi-view fusion as described in claim 1, characterized in that, By improving the YOLO model, a fusion feature recognition model is obtained. The improvement method is to adjust the input layer of the YOLO model to receive multiple single-view visual features, and add a cross-view feature fusion module between the backbone and neck to fuse multiple single-view visual features. Based on the fusion feature recognition model, frame-level fusion and spatial rule judgment are performed on the visual features of each single viewpoint and single frame. The specific process includes: Visual features of the first video frame for each viewpoint Calculate the query vector, key vector, and value vector, where n represents the total number of viewpoints. For each business detection point, determine the attention weights between the primary viewpoint i and other viewpoints j. : in, K represents the key vector of the j-th viewpoint. j The transpose of the matrix; The query vector representing the primary viewpoint i; Attention weights Normalization is performed to obtain normalized weights. ; The fused visual features are calculated based on the normalized weights: in, Indicates the fusion of visual features. The visual features representing the primary visual field i Indicates the adjustment factor; Represents a value vector from other perspectives; Spatial analysis is performed on the fused visual features. If the target position, depth, and occlusion relationship all conform to the preset spatial rules, it is recorded as 1; otherwise, it is recorded as 0. The spatial analysis results of other video frames under each visual path are recorded and summarized as a multi-view result sequence of the fusion feature recognition model.
8. The target recognition processing method based on sliding window and multi-view fusion as described in claim 7, characterized in that, Before performing frame-level fusion of visual features from various single-viewpoints and single frames, the following steps are also included: Align the time of each camera with the time of the NTP server; During video capture, OSD timestamps are added to each video stream and set to a uniform format; At the same frequency, frames are extracted from each video stream to obtain images to be processed. The OSD timestamps on the images are identified to extract time information. Based on the extracted time information, the images are classified and sorted in chronological order, and the group of images with the closest timestamps is selected as a set of images from different perspectives at the same time. Feature extraction is performed on the selected image set to obtain the visual features from various perspectives. .
9. The target recognition processing method based on sliding window and multi-view fusion as described in claim 1, characterized in that, Smoothing is performed on each single-view result sequence and multi-view result sequence separately. The processing includes: Dynamically select the time span based on the business scenario and define a sliding window; Based on the sliding window size, record the corresponding length of the discretized recognition result sequence, represented as follows: ,in ; For each result within the sliding window, the mode is counted. If the mode is unique within the window, the smoothed result is equal to that mode. If there are multiple modes within the window, the mode that appears most frequently is taken as the smoothed result.
10. The target recognition processing method based on sliding window and multi-view fusion as described in claim 1, characterized in that, The method for determining the recognition result of the current video segment is as follows: After smoothing each result sequence, the smoothed result is output. The average or mode of each smoothed result is calculated as the final identification result.