Method and system for multi-camera data backflow based on embedded device

Abstract

The application discloses a kind of multi-path camera data backflow method and system, contain two parts, embedded device and PC end, the application supports on embedded device with USB interface, collect lossless or lossy image, support multiple image formats, support the meta attribute of saving image, support backflow image data by USB interface, for face detection or recognition Intelligent algorithm function processing module, realize statistical analysis or scene reproduction etc..The application also supports simultaneously collecting the data of multiple-path camera, saves the correlation between images from different cameras, for collecting the image corresponding to multiple-path camera at the same time, supports the data collection and data backflow requirement of living body detection algorithm.The application is based on PYTHON tool, realizes the functional requirement of large data collection and backflow, improves the level of automated testing, improves development and testing efficiency.

Description

[0001] This invention is a divisional application of application number 2023109126131, filed on July 25, 2023, entitled "Automatic Testing Method and System for Multi-Channel Camera Data Acquisition and Data Backflow". Technical Field

[0002] This invention relates to the field of computer technology, and in particular to a method and system for reverse data transfer from multiple cameras based on embedded devices. Background Technology

[0003] Embedded devices, such as smart door locks, have an important component called a smart module. The key functional modules in a smart module are face detection, face recognition, and / or liveness detection.

[0004] The training of the algorithm model is typically completed on a personal computer or GPU, and the resulting model is then applied to embedded devices. Additionally, for liveness detection algorithms, it is necessary to acquire images from two or more cameras simultaneously.

[0005] Generally, the face images used to train the algorithm need to be captured from the camera of an embedded device. This can ensure that the algorithm model is consistent with the actual application scenario as much as possible, resulting in better detection and recognition performance.

[0006] In addition, in practical applications, it is necessary to feed facial data back to the intelligent algorithm to statistically analyze the detection or recognition performance of the algorithm model, or to locate any vulnerabilities in the algorithm model.

[0007] Currently, data acquisition typically involves saving camera video data to files. Video formats are generally MJPEG or H.264. This acquisition method has the following drawbacks: 1. The acquired data undergoes compression, resulting in varying degrees of distortion. 2. The acquired data generally requires further classification or recalibration. 3. It cannot simultaneously save data from two or more cameras at the same time, failing to meet the data acquisition requirements of algorithms such as liveness detection.

[0008] Currently, data backfeeding typically uses dedicated equipment (such as HDMI to AHD converters) to convert captured video into virtual camera footage. The drawbacks are: 1. Dedicated equipment incurs additional costs. 2. The compressed video data is distorted. 3. Embedded devices without camera interfaces (such as smart lock modules) cannot use this method. 4. It cannot meet the data backfeeding requirements of liveness detection algorithms.

[0009] Alternatively, there is a data transfer method that transmits video or image data via Ethernet port; however, this method relies on the embedded device having an Ethernet port or Wi-Fi connectivity. It cannot be used for embedded devices without an Ethernet port (such as smart door lock modules). Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for backtracking data from multiple cameras based on embedded devices. This method has low equipment requirements, does not rely on network ports and video interfaces, and supports the requirements for data acquisition and backtracking from multiple cameras.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A method for reverse data transmission from multiple cameras based on an embedded device includes the following steps:

[0013] Step S1: In the camera module of the embedded device, capture YUV data from cameras 1 to N. Images captured by multiple cameras at the same time have the same image ID value, which is used to determine the correlation between images. Compare timestamps to ensure that multiple images were captured at the same time, which is used for liveness detection. Identify the N images corresponding to the liveness detection algorithm as cam1_idM to camN_idM, where cam1 is the image data captured from camera 1, camN is the image data captured from camera N, and idM represents the images captured at the same time point. Save the image's metadata.

[0014] Step S5: Save the image and all image attribute information to a file. The image attribute information is encapsulated into a regular image file name, so that the PC can parse the corresponding image attribute information from the image file name.

[0015] Step S6: Connect the embedded device to the USB interface using a data cable to connect the embedded device to the PC. The user can then operate the embedded device on the PC via ADB.

[0016] Step S7: The PC continuously polls the embedded device to check if the captured images exist, obtains the image list via ADB, and downloads them to the PC sequentially via ADB.

[0017] Step S8: The PC client saves the downloaded image data to the specified folder;

[0018] Step S9: The PC selects images captured at the same time from the data collected by N cameras and uploads them to the embedded device. Using Python, the ID value is parsed from the image file name. That is, the image with ID value M is parsed from camera 1, and the image with ID value M is parsed from camera N. The N parsed images are the image set captured at the same time. The image set is uploaded to the embedded device using ADB.

[0019] Furthermore, the process of acquiring image data from embedded devices on a PC specifically includes the following steps:

[0020] Step a: The PC sends a start recognition command via ADB SHELL;

[0021] Step b: Loop through the embedded device to check if it has successfully acquired and recognized the image, and obtain a list of successfully acquired images;

[0022] Step c: Send an image upload command via ADB to upload the acquired image and save the image to the specified folder. If the folder does not exist, create it.

[0023] Step d: Determine if the number of images acquired is greater than the threshold. If it is greater than the threshold, proceed to step e; otherwise, proceed to step b.

[0024] Step e, End.

[0025] Another objective of this invention is to overcome the shortcomings of the prior art and provide a multi-channel camera data backflow system based on embedded devices, which has low device requirements, does not rely on network ports and video interfaces, and supports multi-channel camera data acquisition and data backflow requirements.

[0026] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0027] A multi-channel camera data backflow system based on an embedded device includes multiple cameras, an embedded device, and a PC.

[0028] Multiple cameras are used to capture raw images;

[0029] The embedded device captures YUV data from cameras 1 to N. Images captured simultaneously by cameras 1 to N have the same image ID value, used to determine the correlation between images. Timestamps are compared to ensure that multiple images were captured at the same time for liveness detection. The N images input to the liveness algorithm are labeled cam1_idM to camN_idM, where cam1 is the image data captured from camera 1, camN is the image data captured from camera N, and idM indicates images captured at the same time point. Meta-attributes of the images are saved. The embedded device performs face detection, obtains the face detection result, and saves the face detection result as image attribute information. It also obtains multiple images from the same time, performs liveness detection, obtains the liveness detection result, and saves the liveness detection score as image attribute information. Finally, it performs face recognition, obtains the face recognition result, and saves the face recognition result as image attribute information. The images and all image attribute information are saved to a file, and the image attribute information is encapsulated into regular image file names, allowing the PC to parse the corresponding image attribute information from the image file names.

[0030] The PC connects to the embedded device's USB interface using a data cable. ADB is used on the PC to operate the embedded device via Python. When acquiring image data, the embedded device is polled to check if any images exist, a list of images is obtained, and each image is downloaded sequentially to the PC. The PC saves the downloaded image data to a specified folder. During data backhaul, images acquired at the same time from N cameras are selected and uploaded to the embedded device. Python is used to parse the ID value from the image file name; for example, the image with ID value M is parsed from camera 1, and the image with ID value M is parsed from camera N. The resulting N images constitute the image set acquired at the same time, which is then uploaded to the embedded device using ADB.

[0031] Furthermore, the specific process for the PC to acquire image data from the embedded device includes the following steps:

[0032] Step a: The PC sends a start recognition command via ADB SHELL;

[0033] Step b: Loop through the embedded device to check if it has successfully acquired and recognized the image, and obtain a list of successfully acquired images;

[0034] Step c: Send an image upload command via ADB to upload the acquired image and save the image to the specified folder. If the folder does not exist, create it.

[0035] Step d: Determine if the number of images acquired is greater than the threshold. If it is greater than the threshold, proceed to step e; otherwise, proceed to step b.

[0036] Step e, End.

[0037] After adopting the above scheme, the multi-camera data backflow method and system of this invention supports the acquisition of lossless or lossy images on embedded devices with USB interfaces, and supports multiple image formats, such as YUV data format (lossless), MJPEG, and JPEG formats. It supports saving image metadata, such as image resolution, image format, image source, or purpose. It supports backflow of image data acquired via USB interface for use in intelligent algorithm processing modules such as face detection or recognition, enabling statistical analysis or scene reproduction. This invention also supports simultaneous acquisition of data from multiple cameras, saving the correlation between images from different cameras, and is used to acquire images from multiple cameras simultaneously, supporting the data acquisition and backflow requirements of liveness detection algorithms. Furthermore, this invention, based on Python tools, fulfills the functional requirements for large-scale data acquisition and backflow, further improving the level of automated testing and increasing development and testing efficiency.

[0038] Compared to existing technologies, this invention has low equipment requirements, does not rely on network ports or video interfaces, and only requires a USB interface; it supports multiple data formats, including lossless and lossy data, to more flexibly meet different development and testing needs; it supports automated testing, reducing implementation costs, saving time, and providing more flexibility in choosing solutions; and it supports multi-channel camera data acquisition, meeting the data acquisition and data backflow requirements of liveness detection algorithms. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the system framework of the present invention.

[0040] Figure 2 This is a schematic diagram of part of the process of the present invention. Detailed Implementation

[0041] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0042] The overall idea of ​​the automatic testing method for multi-channel camera data acquisition and data backflow in this invention is that in embedded devices such as smart door locks, a large number of face images need to be acquired to train intelligent algorithms, such as face detection and face recognition algorithms.

[0043] Generally, the face images used to train the algorithm need to be captured from the camera of the embedded system. This can ensure that the algorithm model is consistent with the actual application scenario as much as possible, resulting in better detection and recognition effects.

[0044] Smart devices, such as door lock modules, typically have a USB interface and support ADB functionality. USB channels are characterized by high bandwidth and fast speed. Therefore, utilizing the USB channel to download images from a camera allows for flexible selection of raw or encoded data, enabling the acquisition of different data formats to meet varying needs.

[0045] In smart locks, to verify the effectiveness of facial recognition, it's necessary to statistically analyze the success and failure probabilities of both a specific large dataset and a set of recognized images. It's also crucial to record which image succeeded or failed.

[0046] Therefore, this invention utilizes the PYHON tool to automatically upload and download images for a large dataset test set, perform face detection or recognition, and then save the face detection or recognition scores and corresponding images to a report. The success or failure rate is statistically analyzed based on specific thresholds to determine whether the algorithm's performance meets requirements, or to locate any defects in the algorithm and assist in identifying those defects.

[0047] Furthermore, liveness detection algorithms require data from multiple cameras simultaneously as input. Therefore, it is necessary to preserve the correlation between data from multiple cameras to meet the synchronization requirements of image data.

[0048] like Figure 1 and Figure 2 As shown, based on the above ideas, this invention discloses an automatic testing method for multi-channel camera data acquisition and data backflow, comprising two parts: an embedded device program (an embedded system) and a PC-side device executing a Python program. The method includes the following steps:

[0049] In the camera module of an embedded device, the raw image is captured. The image format is generally YUV or RGB, etc. It is advisable to directly acquire the raw format image to ensure that the algorithm model is consistent with the actual application scenario, so as to achieve better detection and recognition results. Considering that the raw data format will occupy more storage and bandwidth, the image can also be compressed, such as MJPEG.

[0050] It stores the image's metadata, such as resolution, timestamp, and image ID value. Images captured simultaneously by multiple cameras have the same image ID value, which is used to determine the correlation between images.

[0051] Perform face detection and save the face detection results, such as face coordinates and face detection scores, as image attribute information.

[0052] Multiple images are acquired simultaneously, liveness detection is performed, and the liveness detection score is saved as image attribute information.

[0053] Perform facial recognition, such as obtaining the highest score and the person's name, and save the facial recognition results as image attribute information.

[0054] The image is saved as a file, and the image attribute information is encapsulated into a regular image file name, so that the PC can parse the corresponding image attribute information from the image file name.

[0055] Connect the embedded device to its USB port using a data cable to connect the embedded device to the PC. Users can then operate the embedded device from the PC via ADB, such as running a shell or transferring files.

[0056] During data acquisition, the PC client continuously checks for the existence of images to be acquired, obtains the image list via ADB, and downloads them to the PC client sequentially via ADB.

[0057] The PC client saves the downloaded image data to a specified folder; the folder name includes information such as camera ID, start time of acquisition, and acquisition scene.

[0058] During data backfeeding, the PC obtains an image list from the collected image set and uploads the images sequentially to the embedded device via ADB;

[0059] After the image is uploaded, the PC sends the corresponding detection or recognition command via ADB, waits for completion, and then queries the results.

[0060] Embedded devices execute detection or identification commands and report the results;

[0061] After receiving the reported results on the PC, the image and the corresponding results are saved to an Excel report, and the success or failure probability is calculated.

[0062] Sending commands, querying results, and executing image upload and download commands are all implemented using Python, thereby enabling automatic data acquisition and data backflow based on large datasets.

[0063] When performing liveness detection, the PC client uses multiple images with the same image ID value in the same scene as input.

[0064] like Figure 2 As shown, the process of acquiring and recognizing facial data using the embedded device of this invention specifically includes the following steps:

[0065] Step A: Capture YUV data from cameras 1 to N and compare timestamps to ensure that multiple images were captured at the same time for liveness detection. If data compression is required, you can choose to encode the YUV images as MJPEG, which are lossy images.

[0066] Step B: Perform face detection and obtain the face detection result; if the face detection is successful, proceed to step C; otherwise, proceed to step A.

[0067] Step C: Perform a liveness detection and obtain the liveness detection result; if the liveness detection is successful, proceed to step D; otherwise, proceed to step A.

[0068] Step D: Perform face recognition and obtain the face recognition result; if the face recognition is successful, proceed to step E; otherwise, proceed to step A.

[0069] Step E: Save the image and all image attributes to a file. In embedded devices, if there is no SD device, you can also consider using a memory file system (RAMFS). RAMFS is a memory-based file system that works entirely in RAM. It is a virtual file system that can dynamically create temporary file systems in memory at runtime without using space on the hard disk.

[0070] All recognition scores and image attribute information are encapsulated in a file; specific rules can be customized. The main attribute information includes the following:

[0071]

[0072] The specific process of acquiring image data from an embedded device on a PC includes the following steps:

[0073] Step a: The PC sends a start recognition command via ADB SHELL;

[0074] Step b: Loop through the embedded device to check if it has successfully acquired and recognized the image, and obtain a list of successfully acquired images;

[0075] Step c: Send an image upload command via ADB to upload the acquired image and save the image to the specified folder. If the folder does not exist, create it.

[0076] Step d: Determine if the number of images acquired is greater than the threshold. If yes, proceed to step e; otherwise, proceed to step b.

[0077] Step e, End.

[0078] This method for processing correlations between multi-camera data uses the same ID value to determine image correlation. It is simple to implement, flexible, and applicable even to cases where image data is unrelated. More specifically, during data acquisition, the two images input to the liveness detection algorithm are labeled cam1_id100 and cam2_id100, where cam1 is the image data captured from camera 1 and cam2 is the image data captured from camera 2. id100 indicates an image captured at the same time point. During data backfeeding, images captured at the same time point need to be selected from the data collected from the two cameras and uploaded to the embedded device. At this point, Python is used to parse the ID value from the image file name. For example, if the ID value is 100 from camera 1 and 100 from camera 2, these two images are considered image pairs captured at the same time. ADB is used to upload the image pairs to the embedded device, and the algorithm is executed for detection, thus achieving scene reconstruction. This method is convenient for applications such as algorithm problem localization and registration recognition.

[0079] like Figure 1 As shown, the present invention also discloses an automatic testing system for multi-channel camera data acquisition and data backflow based on an embedded system, which includes multiple cameras 1, an embedded device 2 and a PC 3;

[0080] Multiple cameras 1 are used to capture raw images;

[0081] Embedded device 2 is used to receive raw images from multiple cameras and save the image metadata. Images acquired simultaneously by multiple cameras have the same image ID value, which is used to determine the correlation between images. Embedded device 2 is also used to perform face detection and save the face detection results; perform liveness detection and save the liveness detection score as image attribute information; perform face recognition and save the face recognition result as image attribute information; and save the images as files, with the image attribute information encapsulated into regular image file names, so that the PC can parse the corresponding image attribute information from the image file name. Embedded device 2 is also used to execute detection or recognition commands sent by the PC and report the results.

[0082] The PC connects to the embedded device's USB interface using a data cable. The embedded device is operated via Python on the PC using ADB. During data acquisition, the system polls to check for the existence of images, retrieves the image list via ADB, and downloads them sequentially to the PC. The PC saves the downloaded image data to a designated folder, whose name includes the camera ID, start acquisition time, and acquisition scene. During data backhaul, the PC retrieves the image list from the acquired image set and uploads the images sequentially to the embedded device via ADB. After image upload, the PC issues corresponding detection or recognition commands via ADB, waits for completion, and queries the results. Upon receiving the reported results from the embedded device, the PC saves the images and corresponding results to an Excel report and calculates the success or failure probability.

[0083] The above embodiments and illustrations are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. A method for reverse data transmission from multiple cameras based on embedded devices, characterized in that, Includes the following steps: Step S1: In the camera module of the embedded device, YUV data from cameras 1 to N are captured. Images captured by multiple cameras at the same time have the same image ID value, which is used to determine the correlation between images. Timestamps are compared to ensure that multiple images are captured at the same time, which is used for liveness detection. The N images corresponding to the liveness algorithm are identified as cam1_idM to camN_idM, where cam1 is the image data captured from camera 1, camN is the image data captured from camera N, and idM represents the images captured at the same time point. The meta-attributes of the images are saved. Perform face detection and save the face detection results as image attribute information; Multiple images are acquired simultaneously, liveness detection is performed, and the liveness detection score is saved as image attribute information. Perform face recognition and save the face recognition results as image attribute information; Step S5: Save the image and all image attribute information to a file. The image attribute information is encapsulated into a regular image file name, so that the PC can parse the corresponding image attribute information from the image file name. Step S6: Connect the embedded device to the USB interface using a data cable to connect the embedded device to the PC. The user can then operate the embedded device on the PC via ADB. Step S7: The PC continuously polls the embedded device to check if the captured images exist, obtains the image list via ADB, and downloads them to the PC sequentially via ADB. Step S8: The PC client saves the downloaded image data to the specified folder; Step S9: The PC selects images captured at the same time from the data collected by N cameras and uploads them to the embedded device. Using Python, the ID value is parsed from the image file name. That is, the image with ID value M is parsed from camera 1, and the image with ID value M is parsed from camera N. The N parsed images are the image set captured at the same time. The image set is uploaded to the embedded device using ADB.

2. The method for reverse data transmission from multiple cameras based on embedded devices as described in claim 1, characterized in that: The process of acquiring image data from an embedded device on a PC includes the following steps: Step a: The PC sends a start recognition command via ADB SHELL; Step b: Loop through the embedded device to check if it has successfully acquired and recognized the image, and obtain a list of successfully acquired images; Step c: Send an image upload command via ADB to upload the acquired image and save the image to the specified folder. If the folder does not exist, create it. Step d: Determine if the number of images acquired is greater than the threshold. If it is greater than the threshold, proceed to step e; otherwise, proceed to step b. Step e, End.

3. A multi-channel camera data backflow system based on embedded devices, characterized in that, Includes multiple cameras, embedded devices, and PCs; Multiple cameras are used to capture raw images; The embedded device captures YUV data from cameras 1 to N. Images captured simultaneously by cameras 1 to N have the same image ID value, which is used to determine the correlation between images. Timestamps are compared to ensure that multiple images were captured at the same time, which is used for liveness detection. The N images input to the liveness detection algorithm are labeled as cam1_idM to camN_idM, where cam1 is the image data captured from camera 1, camN is the image data captured from camera N, and idM indicates the images captured at the same time point. The image's metadata is saved. The embedded device is used to perform face detection, obtain face detection results, save face detection results as image attribute information, obtain multiple images simultaneously, perform liveness detection, obtain liveness detection results, and save the liveness detection score as image attribute information, perform face recognition, obtain face recognition results, and save the face recognition results as image attribute information; and save the image and all image attribute information to a file. The image attribute information is encapsulated into a regular image file name, so that the PC can parse the corresponding image attribute information from the image file name. The PC connects to the embedded device's USB interface using a data cable. ADB is used on the PC to operate the embedded device via Python. When acquiring image data, the embedded device is polled to check if any images exist, a list of images is obtained, and each image is downloaded sequentially to the PC. The PC saves the downloaded image data to a specified folder. During data backhaul, images acquired at the same time from N cameras are selected and uploaded to the embedded device. Python is used to parse the ID value from the image file name; for example, the image with ID value M is parsed from camera 1, and the image with ID value M is parsed from camera N. The resulting N images constitute the image set acquired at the same time, which is then uploaded to the embedded device using ADB.

4. The multi-channel camera data backflow system based on embedded devices as described in claim 3, characterized in that: The specific process of acquiring image data from an embedded device on a PC includes the following steps: Step a: The PC sends a start recognition command via ADB SHELL; Step b: Loop through the embedded device to check if it has successfully acquired and recognized the image, and obtain a list of successfully acquired images; Step c: Send an image upload command via ADB to upload the acquired image and save the image to the specified folder. If the folder does not exist, create it. Step d: Determine if the number of images acquired is greater than the threshold. If it is greater than the threshold, proceed to step e; otherwise, proceed to step b. Step e, End.

Images