Layered processing system for edge device video data

Abstract

The invention relates to a hierarchical processing system for edge device video data, and belongs to the technical field. The hierarchical processing system for video acquisition and processing is designed according to the resource characteristics of the edge device for acquiring the video stream, and a set of solution is provided from acquisition, analysis to storage, so that the data processing speed is increased, the data processing cost is reduced, and the better data processing capability is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field, and in particular relates to a layered processing system for edge device video data. Background Art

[0002] Edge computing is a distributed computing framework that allows IoT devices to quickly process data at the edge of the network, moving computing power and storage resources closer to the source of the data. By processing data on edge devices, the reliance on centralized servers is reduced. Edge devices refer to computing devices located at the edge of the network, which usually have computing, storage and communication capabilities. These devices can be smartphones, cameras, sensors, etc., or more complex systems such as medical equipment, self-driving cars, etc. Edge devices are usually located at the source of data generation and can operate independently or work in conjunction with other devices.

[0003] With the development of society, outdoor deployment of cameras for real-time video acquisition plays an important role in security, electricity, water conservancy and other aspects. As a typical stream data, video data plays an important role in the data middle platform and various big data analysis scenarios. The processing of video data includes collection, analysis, visualization, storage and other processes. After the video data is collected, it is used for real-time monitoring of the target or scene, the collected video data is stored in segments, and then used for retrieval and information assistance, and the target information or information of interest contained in the video is analyzed and processed to form structured information. In this process, it is processed through three different processes, and the processing and maintenance work is relatively cumbersome and difficult. According to the above processing characteristics of video data, video data can be regarded as a combination or unity of stream data and batch data.

[0004] Video structuring, that is, the structured processing of video data, extracts key information and makes semantic descriptions through intelligent analysis of the original video. Video is mainly composed of continuous image frames. There is a clear difference between video stream processing and image processing. The core problem is that the command itself has no obvious boundaries and is continuous, making it difficult to directly obtain analysis results like discrete image analysis. The structured analysis of video streams must integrate the analysis results of the target, involving the detection and judgment of the same target, the judgment and fusion of the same type of results, the time and frequency of the results, the reduction of false alarms and missed alarms, and the real-time requirements of processing, that is, real-time (real-time processing, real-time results), accuracy, and moderation. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is to design a hierarchical processing system for video acquisition and processing, so as to improve the data processing speed and reduce the cost of data processing.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention provides a design method for a hierarchical processing system for edge device video data. The edge device has a wireless module and an RJ45 interface. The IP address of the edge device and the IP address of the camera are set to the same local area network segment. The edge device is connected to the camera using a wifi or network cable mode. According to the requirements of video acquisition, real-time video processing, video batch processing, and video services, the system is designed to include:

[0009] Video access layer: obtain real-time video streams on edge devices through the rtsp protocol, onvif protocol or the interface provided by the camera manufacturer;

[0010] Real-time presentation layer: Process the real-time images of the scenes in the video stream as needed, determine the changed image frames, and then transmit the changed image frames to the server in a compressed and encrypted manner according to the established format and requirements;

[0011] Real-time processing layer: Use image analysis algorithms to analyze the changed image frames stored on the server, and send image frames with the target type of interest to the server;

[0012] Batch processing layer: Based on the detection results of the changed image frames in the real-time presentation layer, the changed video segments in the video stream are processed to generate structured target information;

[0013] Service layer: Analyze and count the structured target information, and then organically integrate it with the changed video clips and changed image frames to generate multiple thumbnail videos, providing an information basis for information retrieval and video backtracking on the server side.

[0014] The invention also provides a system designed based on the method.

[0015] (III) Beneficial effects

[0016] The present invention is aimed at edge devices that acquire video streams. According to the resource characteristics of edge devices, a hierarchical processing system for video acquisition and processing is designed, and a set of solutions from acquisition, analysis to storage are proposed, which is committed to improving data processing speed, reducing data processing costs, and achieving better data processing capabilities. DETAILED DESCRIPTION

[0017] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with embodiments.

[0018] Edge devices, such as RK3588, have CPU, memory, GPU and NPU. GPU can be used for conventional image processing, and NPU can be used for deep learning-based calculations. However, compared with personal computers or servers, they still have relatively few computing resources and relatively weak communication resources. Their role is to complete appropriate calculations on the local machine as much as possible, and only transmit as little information or data as possible back to the server. By connecting the edge device to the camera, the video stream can be obtained and processed. The present invention refers to the Lambda architecture for big data processing and performs layered design and processing of video processing on edge devices.

[0019] The present invention provides a hierarchical processing system for edge device video data, and the overall system architecture is designed as follows:

[0020] According to the requirements of video acquisition, real-time video processing, video batch processing, video services, etc., a five-layer system architecture is constructed:

[0021] 1) Video access layer: The edge device is equipped with a wireless module and an RJ45 interface. The IP address of the edge device and the IP address of the camera are set to the same LAN segment. The edge device and the camera are connected using wifi or network cable mode. The video access layer obtains the real-time video stream on the edge device through the rtsp protocol, onvif protocol or the interface provided by the camera manufacturer;

[0022] 2) Real-time presentation layer: Process the real-time images of the scenes in the video stream as needed, determine the changed image frames, and then transmit the changed image frames to the server in a compressed and encrypted manner according to the established format and requirements;

[0023] 3) Real-time processing layer: Use image analysis algorithms to analyze the changed image frames stored on the server, and send image frames with the target type of interest to the server;

[0024] 4) Batch processing layer: Based on the detection results of the changed image frames in the real-time presentation layer, the changed video segments in the video stream are processed to generate structured target information;

[0025] 5) Service layer: Analyze and count the structured target information, and then integrate it with the changed video clips and changed image frames to generate multiple thumbnail videos, providing an information basis for information retrieval and video backtracking on the server side.

[0026] The specific method of processing the real-time image of the scene by the real-time presentation layer is as follows:

[0027] Edge devices generally communicate with the server wirelessly, and returning each frame of image to the server will occupy network resources. However, most of the content of the video stream is fixed or does not need to be paid attention to. Therefore, in order to avoid waste of resources and reduce the storage of invalid information, while considering real-time performance, the following method is used to process the real-time image of the scene:

[0028] Step 1: Prepare n static images without targets or images without abnormalities at different time periods as reference images in advance, and prepare m images for each time period; divide 24 hours into 6 time periods, prepare an image every 4 hours, and prepare 3 images for each time period according to rainy, cloudy, and sunny weather;

[0029] Step 2: Get the current time and image frame. If the current time is an hour, return the current image to the server. If it is not an hour, jump to Step 3.

[0030] Step 3: Compare the image frame with the pre-prepared static image every 1 second based on the frame difference method. If there is no change, return the Json information {target:0; changed:0}; if there is a change, jump to Step 4;

[0031] Step 4: If there is a change, use the SVM-based data classification method to perform binary classification confirmation on the current image frame. If there is a change, return the current frame to the server and jump to Step 5; otherwise, jump to Step 2;

[0032] Step 5: Get the current image frame and compare it with the previous frame using the frame difference method. If there is no change, do not respond; if there is a change, jump to Step 6;

[0033] Step 6: Use the SVM-based data classification method to compare the current image frame with the static image. If there is no abnormality, jump to Step 2; if the target changes, jump to Step 4.

[0034] Contents of the real-time processing layer:

[0035] When the image frame changes, the target detection and analysis algorithm is used to analyze the specific situation of the change. The object of processing is the changed image frame, and the content of analysis is mainly whether the target type of concern appears. In order to reduce false alarms and missed alarms, a multi-detection method fusion method is used to analyze the changed image frame:

[0036] Step 1: Read the changed image frames and the image frames with an interval of 2 seconds;

[0037] Step 2: Use the target detection method based on CascadeClassifier, the target detection method based on yolov8, and the method based on Faster R-CNN to perform target detection on the same frame image;

[0038] Step 3: Based on the detection results, vote on the type, position, and quantity to obtain the results about the target, position, and quantity;

[0039] Step 4: If the target type is the attention type, return the current image frame to the server, otherwise no subsequent processing is performed;

[0040] If the target type is a focus type, it is further determined whether there is an electronic fence and whether the target has entered the electronic fence. If it has entered the electronic fence, the alarm mechanism is triggered and the image frame is sent back to the server as needed; if it has not entered, the alarm mechanism is not triggered and no subsequent processing is performed.

[0041] Contents of the batch layer:

[0042] The acquired video is streaming data. Generally, for the convenience of analysis, the video slices are saved as video segments at fixed intervals, but this method will lead to the fragmentation of analysis. Here, according to the detection results of the changed image frames in the real-time presentation layer, the images in the changed video segments are sliced ​​to obtain the sequence of initial frame-no change-image frame-change-image frame... and save it. Since the unchanged video segments have no analysis value, only the changed video segments are analyzed. The analysis of the video segments is carried out when resources are idle, such as at night. The batch layer processes the video segments as follows:

[0043] Step 1: Get a video clip that has changed;

[0044] Step 2: Detect objects in the video clips and classify them into three types: people, animals, and vehicles. Record the start and end time of the objects respectively.

[0045] Step 3: For human-type video clips, perform posture recognition based on Open Pose and record the start and end time of the posture category; for vehicle-type video clips, identify vehicle type, license plate number, vehicle color and other information.

[0046] Contents of the service layer:

[0047] Summarize and integrate the analysis results to generate higher-level and more complete information.

[0048] Step 1: Integrate the personnel information. Process the structured target information obtained from the batch processing layer to obtain the feature vectors of the targets. Each video segment with personnel corresponds to one or more feature vectors of the targets (personnel). Traverse all video segments with personnel, match the feature vectors to achieve person re-identification, and count information such as the number of times a person appears on the same day, the start and end times each time, and the direction of entering and exiting the video. The method for matching the feature vectors is as follows:

[0049] Step 11: Sum and average each component of the feature vector of a certain target in each video segment with personnel, and sort the targets by serial number. Assume the serial numbers range from 1 to n, and let i = 1;

[0050] Step 12: Extract the feature vector of the target with serial number i, traverse the feature vectors from i + 1 to n, and calculate the similarity of the feature vectors one by one using the Euclidean distance. If the similarity of two feature vectors is less than the set threshold, it is considered that these two feature vectors contain the same target, and this feature vector is deleted from the feature vectors to be matched;

[0051] Step 13: Let i = i + 1. If i < n, that is, the feature vector of the target with serial number i exists, jump to Step 12. If i = n, end the matching process.

[0052] Step 2: Integrate the vehicle information. Taking the license plate and vehicle type as the analysis dimensions, count information such as the number of times a vehicle appears currently, the start and end times each time, and the direction of entering and exiting the video;

[0053] Step 3: Summarize the personnel and vehicle information, and establish overall statistical information such as the number, frequency, and time of personnel on the same day;

[0054] Step 4: Associate the structured information of personnel and vehicles obtained from the analysis with the video segments that have changed and the image frames that have changed, and generate the overall thumbnail video of the day, the overall thumbnail video of personnel, the overall thumbnail video of vehicles, the thumbnail video of a certain person on the same day, and the thumbnail video of a certain vehicle or a certain type of vehicle on the same day, and store them on the edge device or return them to the server.

[0055] Method for generating the overall thumbnail video of the day: Concatenate the video segments where the targets appear in chronological order, and then recompose the video at a frame rate of 1 frame / second;

[0056] Method for generating the overall thumbnail video of personnel: Concatenate the video segments where personnel appear in chronological order, and then recompose the video at a frame rate of 1 frame / second;

[0057] The overall thumbnail video generation method of the vehicle is as follows: the video clips in which the vehicle appears are spliced ​​together in chronological order, and then reassembled into a video at a frame rate of 1 frame / second;

[0058] Method for generating a thumbnail video of a person on the day: splice the video clips of the person in chronological order, and then reassemble the video at a frame rate of 1 frame / second;

[0059] A method for generating a thumbnail video of a vehicle on the day is as follows: the video clips in which the vehicle appears are spliced ​​together in chronological order, and then reassembled into a video at a frame rate of 1 frame per second;

[0060] A method for generating a thumbnail video of a certain type of vehicle on the day is as follows: video clips in which the type of vehicle appears are spliced ​​together in chronological order, and then reassembled into a video at a frame rate of 1 frame per second.

[0061] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A design method for a hierarchical processing system for edge device video data, characterized in that: The edge device has a wireless module and an RJ45 interface. The IP address of the edge device and the IP address of the camera are set to the same LAN segment. The edge device uses Wi-Fi or Ethernet mode to connect to the camera. According to the requirements of video acquisition, real-time video processing, video batch processing, and video services, the system is designed to include: Video access layer: obtain real-time video streams on edge devices through the rtsp protocol, onvif protocol or the interface provided by the camera manufacturer; Real-time presentation layer: Process the real-time images of the scenes in the video stream as needed, determine the changed image frames, and then transmit the changed image frames to the server in a compressed and encrypted manner according to the established format and requirements; Real-time processing layer: Use image analysis algorithms to analyze the changed image frames stored on the server, and send image frames with the target type of interest to the server; Batch processing layer: Based on the detection results of the changed image frames in the real-time presentation layer, the changed video segments in the video stream are processed to generate structured target information; Service layer: Analyze and count the structured target information, and then organically integrate it with the changed video clips and changed image frames to generate multiple thumbnail videos, providing an information basis for information retrieval and video backtracking on the server side.

2. The method according to claim 1, characterized in that The specific method of the real-time presentation layer for processing the real-time image of the scene is as follows: Step 11: Prepare n static images without targets or images without abnormalities at different time periods as reference images in advance, and prepare m images for each time period; divide 24 hours into 6 time periods, prepare an image every 4 hours, and prepare 3 images for each time period according to rainy, cloudy, and sunny weather; Step 12: Get the current time and image frame. If the current time is an hour, return the current image to the server. If it is not an hour, jump to Step 3. Step 13: Compare the image frame with the pre-prepared static image every 1 second based on the frame difference method. If there is no change, return the Json information {target:0; changed:0}; if there is a change, jump to Step 14; Step 14: If there is a change, use the SVM-based data classification method to perform binary classification confirmation on the current image frame. If there is a change, return the current frame to the server and jump to Step 15; otherwise, jump to Step 12; Step 15: Get the current image frame and compare it with the previous frame using the frame difference method. If there is no change, no response is made. If there is a change, jump to Step 16; Step 16: Use the SVM-based data classification method to compare the current image frame with the static image. If there is no abnormality, jump to Step 12; If the target changes, jump to Step 4.

3. The method according to claim 1, characterized in that When the image frame changes, the real-time processing layer uses a target detection and analysis algorithm to analyze the specific situation of the change. The object of processing is the changed image frame, and the content of analysis is whether the target type of interest appears.

4. The method according to claim 2, characterized in that The real-time processing layer uses a fusion of multiple detection methods to analyze the changed image frames: Step 21: Read the changed image frame; Step 22: Use the target detection method based on CascadeClassifier, the target detection method based on yolov8, and the method based on Faster R-CNN to perform target detection on the same frame image; Step 23: Based on the detection results, vote on the type, position, and quantity to obtain the results about the target, position, and quantity; Step 24: If the target type is the attention type, return the current image frame to the server, otherwise no subsequent processing is performed.

5. The method according to claim 4, characterized in that In Step 24, if the target type is a focus type, it is further determined whether there is an electronic fence and whether the target has entered the electronic fence. If it has entered the electronic fence, the alarm mechanism is triggered, an alarm is issued, and the image frame is sent back to the server as needed; if it has not entered, the alarm mechanism is not triggered and no subsequent processing is performed.

6. The method according to claim 4, characterized in that The batch processing layer also slices the images in the changed video clips according to the detection results of the changed image frames in the real-time presentation layer, and obtains a sequence of initial frame-no change-image frame-change-image frame for storage.

7. The method according to claim 4, characterized in that The batch layer processes the video clips as follows: Step 31: Get a certain video segment that has changed; Step 32: Perform target detection on the video clips, classify them into three types: people, animals, and vehicles, and record the start time and end time of the target respectively; Step 33: For human-type video clips, perform posture recognition based on Open Pose and record the start and end time of the posture category; for vehicle-type video clips, identify the vehicle type, license plate number, and vehicle color information.

8. The method according to claim 6, characterized in that The batch layer processes video clips during resource idle periods.

9. The method according to claim 7, characterized in that The service layer performs the following steps: Step 41: Integrate personnel information: Process the structured target information obtained by the batch layer to obtain the target feature vector. Each video clip with a person appears corresponds to one or more target feature vectors. Traverse all video clips with a person appearing, match the feature vectors, realize person re-identification, and count the number of times the person appears on the day, the start and end time of each time, and the direction information of entering and leaving the video. The method for matching the feature vectors is: Step 411: sum and average the components of the feature vector of a target in each video clip where a person appears, and sort the targets by sequence number; assuming that the sequence number is from 1 to n, let i=1; Step 412: Take out the feature vector of the target with serial number i, traverse the feature vectors from i+1 to n, and calculate the feature vector similarity one by one using the Euclidean distance. If there are two feature vectors whose similarity is less than the set threshold, it is considered that the two feature vectors contain the same target, and the feature vector is deleted from the feature vectors to be matched; Step 413: Let \(i = i + 1\). If \(i < n\), that is, the feature vector of the target with serial number \(i\) exists, then jump to Step 412. If \(i = n\), end the matching process; Step 42: Integrate vehicle information: Taking license plate and vehicle type as the analysis dimensions, count the current occurrence times of the vehicle, the start and end times each time, and the direction information of entering and exiting the video; Step 43: Summarize the personnel and vehicle information to obtain the overall statistical information of the number, frequency, and time of personnel on the same day; Step 4: Associate the structured information of personnel and vehicles obtained from the analysis with the changed video segments and changed image frames to generate the overall thumbnail video of the day, the overall thumbnail video of personnel, the overall thumbnail video of vehicles, the thumbnail video of a certain person on the same day, and the thumbnail video of a certain vehicle or a certain type of vehicle on the same day, and store them on the edge device or return them to the server; Method for generating the overall thumbnail video of the day: Concatenate the video segments where the target appears in chronological order, and then recompose the video at a frame rate of 1 frame / second; Method for generating the overall thumbnail video of personnel: Concatenate the video segments where personnel appear in chronological order, and then recompose the video at a frame rate of 1 frame / second; Method for generating the overall thumbnail video of vehicles: Concatenate the video segments where vehicles appear in chronological order, and then recompose the video at a frame rate of 1 frame / second; Method for generating the thumbnail video of a certain person on the same day: Concatenate the video segments where the person appears in chronological order, and then recompose the video at a frame rate of 1 frame / second; Method for generating the thumbnail video of a certain vehicle on the same day: Concatenate the video segments where the vehicle appears in chronological order, and then recompose the video at a frame rate of 1 frame / second; Method for generating the thumbnail video of a certain type of vehicle on the same day: Concatenate the video segments where the type of vehicle appears in chronological order, and then recompose the video at a frame rate of 1 frame / second.

10. A system designed based on the method according to any one of claims 1 to 9.