A method and apparatus for real-time dynamic analysis of streaming picture data

Abstract

The application relates to the technical field of data processing, and specifically provides a method for real-time dynamic analysis of streaming picture data, which comprises the following steps: firstly, deploying a multi-camera device, calling an SDK interface of c++ language of the device for use, further encapsulating a required use method into a dynamic library, and then performing compilation and deployment calling to complete encapsulation of the SDK dynamic library; secondly, performing multi-camera integration configuration and video push-pull streaming, wherein the video push-pull streaming operation process queue is used to ensure the stability of the video picture frame number; and finally, performing data analysis and data reasoning, and performing page display on the front end. Compared with the prior art, the application can comprehensively observe the production line dynamics, perform 360-degree product defect detection, track products to ensure that no defects are missed, and dynamically regulate the equipment on the front end, so that the platform is more intelligent.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically providing a method and apparatus for real-time dynamic analysis of streaming image data. Background Technology

[0002] With the development of artificial intelligence technology, single network cameras can only perform rough human body labeling or object recognition in terms of detection. If there is a need for detailed analysis of object surfaces, multiple industrial cameras need to be deployed to comprehensively recognize, analyze and process the images.

[0003] The most fundamental function of industrial cameras is to convert light signals into ordered electrical signals, resulting in higher image stability, transmission capabilities, and anti-interference abilities. Their performance directly affects not only image resolution and quality but also the efficiency of the entire production process.

[0004] In the 4G era, limited by transmission speed and other performance constraints, only ordinary quality images could be transmitted with latency, preventing real-time communication between the front-end and back-end. Now, by incorporating Quectel's 5G modules, industrial cameras can successfully leverage 5G technology, achieving a performance breakthrough. Industrial cameras are a key component in machine vision systems; the performance of their internal image sensor chips and the choice of camera interfaces significantly impact the quality of image acquisition, and even the speed of image transmission and processing. Summary of the Invention

[0005] This invention addresses the shortcomings of the prior art by providing a highly practical method for real-time dynamic analysis of streaming image data.

[0006] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable device for real-time dynamic analysis of streaming image data.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] A method for real-time dynamic analysis of streaming image data first involves deploying multiple camera devices, calling the device's C++ SDK interface, further encapsulating the required usage methods into a dynamic library, compiling and deploying it to complete the encapsulation of the SDK dynamic library.

[0009] Then, multi-camera integration configuration and video push-pull streaming are performed, where images in the video push-pull streaming operation process queue are used to ensure stable video frame rate.

[0010] Finally, data analysis and reasoning are performed, and the data is then displayed on the front end of the page.

[0011] Furthermore, the multi-camera integrated configuration enables integrated configuration when connecting multiple cameras. It achieves multi-camera connection through multiple processes, configures camera attributes based on each process ID, and controls camera start and stop.

[0012] Furthermore, the video push-pull streaming unit includes a pull-stream submodule, a frame submodule, and a push-stream submodule. The pull-stream submodule performs image acquisition, maintains a connection with the camera device, calls the multi-camera device SDK to acquire images, places the images in the process channel according to the timestamp, and controls the rate at which the images enter the process channel according to the camera frame rate.

[0013] Furthermore, the labeling submodule performs image labeling, extracts the image from the process channel, adds the labeled information to the image, and then re-places the image in the process channel according to the timestamp.

[0014] Furthermore, the streaming submodule pushes the images to the signaling server. Using nginx or zlmediakit, the HTTP-FLV video streaming protocol is configured for easy display in a browser. Images are retrieved from the process channel according to their timestamps and pushed to the signaling server to form a video stream. The front-end page can access the streaming address to see the multi-camera images.

[0015] Furthermore, the data analysis involves model training, image recognition, and information feedback. The algorithm is encapsulated into a dynamic library using C++, and the image data in the process queue is sent to the data analysis unit. After the algorithm calculates the data, it returns bounding box data. The bounding box submodule then processes and displays the image using the data.

[0016] Furthermore, data inference is a middleware unit between the video push-pull streaming unit and the data analysis unit. Before the images in the process image queue are pushed to the signaling server, they need to go through the image processing of the data inference unit and the image analysis of the data analysis unit.

[0017] To further determine whether the image has value for analysis, the data is analyzed and then returned to the video push-pull stream unit. After obtaining valuable data, the video push-pull stream unit will redraw the image data and push the new image data to the signaling server for further display on the front end.

[0018] An apparatus for real-time dynamic analysis of streaming image data includes: at least one memory and at least one processor;

[0019] The at least one memory is used to store a machine-readable program;

[0020] The at least one processor is used to call the machine-readable program to execute a method and apparatus for real-time dynamic analysis of streaming image data.

[0021] Compared with existing technologies, the method and apparatus for real-time dynamic analysis of streaming image data of the present invention have the following outstanding advantages:

[0022] The deployment of the multi-camera device of this invention targets the details of the sample being tested, making it more suitable for quality inspection and analysis projects in factories. The customization of the device and the subsequent integration of multiple cameras, including the use of the front-end page, make the entire device more aligned with artificial intelligence.

[0023] This is an AI platform that is more suitable for deployment on industrial production lines. While ensuring the accuracy of quality inspection, it can observe the dynamics of the production line in all aspects, perform 360-degree product defect detection, track products to ensure that no inspection is missed, and the front-end page dynamically controls the equipment, making the platform more intelligent to use. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Appendix Figure 1 This is a flowchart illustrating a method for real-time dynamic analysis of streaming image data. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] The following is a preferred embodiment:

[0028] like Figure 1 As shown in this embodiment, a method for real-time dynamic analysis of streaming image data is characterized by the following steps: First, a multi-camera device is deployed, and the device's C++ language SDK interface is called for use. The required usage methods are further encapsulated into a dynamic library, and then compiled and deployed for use, thus completing the encapsulation of the SDK dynamic library.

[0029] Then, multi-camera integration configuration and video push-pull streaming are performed, where images in the video push-pull streaming operation process queue are used to ensure stable video frame rate.

[0030] Finally, data analysis and reasoning are performed, and the data is then displayed on the front end of the page.

[0031] The deployment of multi-camera equipment includes the customization of multi-camera equipment devices, the design of light sources, and the installation of the number of cameras according to the items being inspected. The customization of the devices should ensure that the items being inspected can pass through without damage or obstruction. The placement of the light sources should ensure that the cameras provide sufficient lighting without affecting the exposure time, and there should be no uneven lighting on the items being inspected. The number of multi-camera devices deployed is generally 3, each responsible for a 120-degree angle, to ensure that the items being inspected are within a ring without blind spots and without repeated inspection of defects.

[0032] The SDK dynamic library is used to establish connections between the server and multiple camera devices. Whether it is a giga industrial camera or a multi-sensor system, establishing an automated connection requires calling the fixed-package SDK interface of the device for connection, parameter tuning, and use. In order to facilitate system calls and take server performance into consideration.

[0033] Multi-camera integrated configuration is used for integrated configuration design when connecting multiple cameras. It realizes the connection of multiple cameras through multiple processes, configures camera attributes according to each process number, and controls the start and stop of a certain camera. Generally, it ensures that the attribute configuration of each frame is the same for image acquisition. During integration, it can dynamically control which camera displays the video stream and shuts it down.

[0034] The video push-pull streaming unit includes a pull submodule, a frame submodule, and a push submodule. The pull submodule acquires images, maintains a connection with the camera device, calls the multi-camera device SDK to acquire images, and places the images in the process channel according to the timestamp. The rate at which the images enter the process channel is controlled according to the camera frame rate.

[0035] The labeling submodule performs image labeling, extracts the image from the process channel, adds the labeled information to the image, and then re-places the image in the process channel according to the timestamp.

[0036] The push streaming submodule pushes images to the signaling server. It uses nginx or zlmediakit and configures the http-flv video streaming protocol for easy display in the browser. It retrieves images from the process channel according to their timestamps and pushes them to the signaling server to form a video stream. The front-end page can access the pull streaming address to see the multi-camera images.

[0037] Data analysis refers to the core algorithm, which relies on the performance of domestically produced circuit boards for model training, image recognition, and information feedback. It is encapsulated in C++ as a dynamic library for algorithm calls, greatly improving algorithm performance. The core functionality involves sending image data from the process queue to the data analysis unit, where the algorithm calculates and returns bounding box data. The bounding box submodule then further processes and displays the images.

[0038] Data inference is a middleware unit situated between the video push / pull streaming unit and the data analysis unit. Before being pushed to the signaling server, images in the process image queue undergo image processing by the data inference unit and image analysis by the data analysis unit to further determine their analytical value. After data analysis, the data is returned to the video push / pull streaming unit. Once the video push / pull streaming unit obtains valuable data, it redraws the image data and pushes the new image data to the signaling server for further display on the front end.

[0039] Once the entire project system is built and the physical machines, conveyor belts, and quality-inspected goods are deployed, the entire process appears as follows: after the system starts running, objects enter the camera container evenly according to the conveyor belt speed for quality inspection. During this process, the camera captures images from multiple angles according to the quality inspection frame rate requirements. The backend data stream is then displayed on the frontend page, and users can log in and adjust the parameters to view the video stream from multiple cameras.

[0040] First, there's the deployment of the machines. To ensure that the physical speed on the conveyor belt is uniform, considering that the conveyor belt may not be uniform due to minor obstacles and acceleration / deceleration, and the multi-camera system needs to automatically adjust the frame rate according to the conveyor belt speed to ensure a stable display effect, a real-time speed detection device needs to be added to the multi-camera container to dynamically detect the current uniform speed of the items, so as to ensure that the subsequent algorithm will not have errors due to uneven item speed when tracking repeated items.

[0041] Additionally, a dynamic camera focus adjustment device needs to be deployed during machine deployment. After multiple cameras are fixed inside a customized container, the focus of the industrial camera device can only be adjusted by physical means. Software generally does not support camera focus adjustment. Therefore, based on our needs, we need to design an adjustment device to allow manual focus adjustment on the page according to the real-time streaming image. This ensures that the image is clear and accurate while also ensuring the convenience of adjustment, preventing the need for focus adjustment when the container device changes in the scene environment or the size of the object.

[0042] Additionally, lighting design is required inside the multi-camera container. Since the container itself is unlit, maintaining image brightness and a stable exposure time for the cameras, adjusted according to the conveyor belt speed, is crucial to prevent blurring even at high speeds. Placing two appropriately sized apertures in the center of the multi-camera system, allowing cylindrical objects like conduits to pass through, ensures clear visibility of objects within the aperture, uniform lighting, and bright, easily viewable images.

[0043] After deploying the multi-camera container equipment, use the dedicated software of the industrial camera SDK to connect the equipment, modify the device IP, ensure that the device network port and the server network port are consistent, and manually adjust the page parameters, including exposure time, general frame rate, and field of view brightness, to ensure that the image is clear before connecting to the server.

[0044] Using camera devices requires the manufacturer's SDK. Commonly used interfaces include establishing a single connection channel for the camera, passing in modifiable camera parameters, and receiving image streams. The process initializes the camera connection, passes in the required parameters, and then repeatedly calls the image stream interface to acquire images. The camera returns the currently captured image to the interface at time intervals, thus enabling image acquisition. In a multi-camera deployment, different processes and channels acquire image information from different fields of view.

[0045] When integrating and configuring multi-camera devices, a global configuration is required based on the project. Different cameras are configured with relevant usage parameters according to different needs, and the activation and deactivation of multiple cameras are controlled under the global configuration. For example, in the case of dynamic focus adjustment, typically one camera is displayed on one page for adjustment, and the adjustment parameters are dynamically configured to ensure that the focus of each camera has unique adjustment.

[0046] In a multi-camera deployment environment, each camera in the project layer opens a separate process channel to perform image acquisition, image framing, and video streaming. Multiple threads are started concurrently under the current process channel to perform image acquisition, image framing, and video streaming. The entire process is also dependent on the image process channel. Three threads operate on the image process channel concurrently. The storage and retrieval of each image are performed based on a fixed timestamp at the time of storage, ensuring that the entire process is safely operated on the image process channel under the control of thread locks, protecting the normal streaming of the video stream.

[0047] After the camera opens the process channel, each process opens a corresponding sub-process for data inference, and the corresponding data inference opens a sub-process for data analysis. The camera's process channel also opens a separate data receiving channel to receive valid data after the image inference and analysis are completed, which is then used by the frame thread to re-edit the image.

[0048] An apparatus for real-time dynamic analysis of streaming image data includes: at least one memory and at least one processor;

[0049] The at least one memory is used to store a machine-readable program;

[0050] The at least one processor is used to call the machine-readable program to execute a method and apparatus for real-time dynamic analysis of streaming image data.

[0051] The specific embodiments described above are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the specific embodiments described above. Any appropriate changes or substitutions made by a person skilled in the art that conform to the claims of the present invention regarding a method and apparatus for real-time dynamic analysis of streaming image data should fall within the patent protection scope of the present invention.

[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time dynamic analysis of streaming image data, characterized in that... First, deploy the multi-camera device, call the device's C++ SDK interface, further encapsulate the required usage methods into a dynamic library, compile and deploy it, and complete the encapsulation of the SDK dynamic library. Then, multi-camera integration configuration and video push-pull streaming units are performed. The video push-pull streaming units operate on images in the process queue to ensure stable video frame rates. Finally, data analysis and reasoning are performed, and the data is displayed on the front end of the page. The video push-pull streaming unit includes a pull-stream submodule, a frame-marking submodule, and a push-stream submodule. The pull-stream submodule acquires images, maintains a connection with the camera device, calls the multi-camera device SDK to acquire images, places the images in the process channel according to the timestamp, and controls the rate at which the images enter the process channel according to the camera frame rate. The labeling submodule performs image labeling, extracts the image from the process channel, adds the labeled information to the image, and then re-places the image in the process channel according to the timestamp. The push streaming submodule pushes the images to the signaling server. Using nginx or zlmediakit, the HTTP-FLV video streaming protocol is configured for easy display in the browser. Images are retrieved from the process channel according to the timestamp and pushed to the signaling server to form a video stream. The front-end page can access the pull streaming address to see the multi-camera images. The data analysis involves model training, image recognition, and information feedback. It uses C++ to encapsulate the algorithm into a dynamic library for calling the algorithm. The image data in the process queue is sent to the data analysis unit, which calculates the data and returns bounding box data. The bounding box submodule then uses the data to further process and display the image. Data inference is a middleware unit between the video push-pull streaming unit and the data analysis unit. Before the images in the process image queue are pushed to the signaling server, they need to go through the image processing of the data inference unit and the image analysis of the data analysis unit. To further determine whether the image has value for analysis, the data is analyzed and then returned to the video push-pull stream unit. After obtaining valuable data, the video push-pull stream unit will redraw the image data and push the new image data to the signaling server for further display on the front end.

2. The method for real-time dynamic analysis of streaming image data according to claim 1, characterized in that, Multi-camera integrated configuration enables integrated configuration when connecting multiple cameras. It achieves multi-camera connection through multiple processes, configures camera attributes based on each process ID, and controls camera start and stop.

3. A device for real-time dynamic analysis of streaming image data, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to execute the method according to any one of claims 1 to 2.

Images