Low-delay quick-response industrial camera image processing method and storage medium

Abstract

The invention relates to a low-delay quick-response industrial camera image processing method and a storage medium, an industrial camera image processing system comprises an industrial camera and a video coding module, and the industrial camera internally comprises an image acquisition module and an algorithm recognition module. A first output port of the image acquisition module is connected with an input end of the video coding module to form a first transmission path, and a second output port of the image acquisition module is connected with an input end of the algorithm identification module to form a second transmission path; wherein the image acquisition module is used for performing image acquisition on a surrounding environment to generate an initial image; the video encoding module obtains an initial image through the first transmission path, encodes the initial image, and generates an encoded video stream; the algorithm identification module obtains the initial image through the second transmission path, carries out algorithm identification based on the initial image, and generates an algorithm identification result. According to the invention, the signal output delay is reduced, and the response speed of the industrial camera image processing system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial camera image processing, in particular to an industrial camera image processing method with low latency and fast response and a storage medium. BACKGROUND

[0002] With the development of industrial automation, more and more mechanical equipment is running in the production environment. In some dangerous areas, such as the working area of a mechanical arm, the intrusion of personnel can cause serious safety accidents. Therefore, it is necessary to set up a safety protection system to monitor the intrusion of personnel in real time and take control measures in time. At present, industrial cameras are mainly used to obtain images, and image processing and algorithm analysis are used to detect personnel intrusion.

[0003] Figure 1 For the workflow diagram of the industrial camera image processing system based on the FFmpeg (Fast Forward Moving Picture Experts Group) framework in the related art, the system first acquires images by an industrial camera and outputs them, encodes the images into videos and saves them to a switch, and then other devices obtain the saved image data from the switch, decode the videos, and then perform video streaming and algorithm recognition. Among them, the video is pushed to the client for users to watch the production environment. The final purpose of algorithm recognition is to output signals to the mechanical equipment to control its running state. This architecture causes the image to be pushed to the algorithm recognition device after being acquired, encoded and decoded, and multiple hardware modules are involved, resulting in a long signal output delay, while the industrial safety protection scene requires fast response capability.

[0004] At present, there is no effective solution to the problem of long signal output delay in industrial automation image processing. SUMMARY

[0005] Therefore, it is necessary to provide an industrial camera image processing method with low latency and fast response and a storage medium to solve the above technical problems.

[0006] In a first aspect, the present application provides an industrial camera image processing method applied to an industrial camera image processing system, wherein the industrial camera image processing system comprises an industrial camera and a video encoding module, the industrial camera internally comprises an image acquisition module and an algorithm recognition module, a first output port of the image acquisition module is connected with an input end of the video encoding module and constitutes a first transmission path, and a second output port of the image acquisition module is connected with an input end of the algorithm recognition module and constitutes a second transmission path; the method comprises:

[0007] The image acquisition module acquires images of the surrounding environment to generate an initial image;

[0008] The video encoding module acquires the initial image through the first transmission path, encodes the initial image, and generates an encoded video stream;

[0009] The algorithm identification module acquires the initial image through the second transmission path, and performs algorithm identification based on the initial image to generate an algorithm identification result.

[0010] In one of the embodiments, the algorithm identification module comprises a preprocessing unit, a cache area management unit and an algorithm unit connected in sequence; the algorithm identification module performs algorithm identification based on the initial image to generate an algorithm identification result, comprising:

[0011] The preprocessing unit acquires the initial image through a zero-copy mode, pre-processes the initial image, and generates a pre-processed image;

[0012] The cache area management unit caches the pre-processed image;

[0013] The algorithm unit reads the pre-processed image from the cache area management unit, performs algorithm identification on the pre-processed image, and generates the algorithm identification result.

[0014] In one of the embodiments, the preprocessing unit pre-processes the initial image to generate a pre-processed image, comprising:

[0015] The preprocessing unit performs scaling processing and / or format conversion on the initial image to obtain the pre-processed image.

[0016] In one of the embodiments, the cache area management unit caches the pre-processed image, comprising:

[0017] The cache area management unit receives the pre-processed image and caches the pre-processed image to a corresponding queue, wherein the queue comprises an RGB frame pushing queue, an OSD processing queue and a scaling processing queue.

[0018] In one of the embodiments, after the cache area management unit receives the pre-processed image and caches the pre-processed image to a corresponding queue, the method further comprises:

[0019] Acquiring a mutual exclusion lock;

[0020] Monitoring the current capacity in each of the queues;

[0021] When it is monitored that the current capacity of any one of the queues exceeds the corresponding capacity limit value, discarding the oldest frame in the queue;

[0022] Releasing the mutual exclusion lock.

[0023] In one embodiment, when it is detected that the current capacity of any queue exceeds its corresponding capacity limit, the method further includes:

[0024] Determine whether the current capacity of the queue exceeds its capacity limit by N times;

[0025] If the current capacity of the queue exceeds N times its capacity limit, a warning log will be output to record that the queue is severely congested.

[0026] In one embodiment, the algorithm unit reads the preprocessed image from the cache management unit, performs algorithm recognition on the preprocessed image, and generates the algorithm recognition result, including:

[0027] The algorithm unit reads RGB frames from the RGB frame push queue of the buffer management unit;

[0028] The algorithm unit checks whether a user callback function has been registered;

[0029] If the user callback function has been registered in the algorithm unit, then the user callback function is invoked to pass the RGB frame;

[0030] Obtain the return value of the user callback function, and obtain the algorithm recognition result based on the return value.

[0031] Secondly, this application provides an industrial camera, including: an image acquisition module and an algorithm recognition module, wherein a first output port of the image acquisition module is connected to the input end of a video encoding module to form a first transmission path, and a second output port of the image acquisition module is connected to the input end of the algorithm recognition module to form a second transmission path;

[0032] The image acquisition module is used to acquire images of the surrounding environment and generate an initial image.

[0033] The video encoding module is used to acquire the initial image through the first transmission path, encode the initial image, and generate an encoded video stream.

[0034] The algorithm recognition module is used to acquire the initial image through the second transmission path, and perform algorithm recognition based on the initial image to generate algorithm recognition results.

[0035] Thirdly, this application provides an industrial camera image processing system, including: a video encoding module and the industrial camera described in the second aspect above, wherein the video encoding module and the industrial camera are connected.

[0036] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0037] The aforementioned industrial camera image processing method, industrial camera, system, and storage medium integrate the image acquisition module and algorithm recognition module within the industrial camera. The algorithm recognition module and video encoding module are connected to the image acquisition module, forming two transmission paths. This allows video encoding and algorithm recognition to be processed in parallel. When image data reaches the algorithm recognition module, it does not need to undergo multiple format conversions and encoding / decoding processes, reducing signal output latency and improving the response speed of the industrial camera image processing system. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the workflow of an industrial camera image processing system based on the FFmpeg framework in related technologies.

[0039] Figure 2 This is a structural block diagram of an industrial camera image processing system in one embodiment;

[0040] Figure 3 This is a flowchart illustrating an industrial camera image processing method in one embodiment;

[0041] Figure 4 This is a block diagram of an industrial camera image processing system in another embodiment;

[0042] Figure 5 This is a flowchart illustrating the internal operation of the algorithm recognition module in one embodiment;

[0043] Figure 6 Here is a flowchart of the queue management process of the buffer management unit in one embodiment;

[0044] Figure 7 This is a schematic diagram of the architecture of an industrial camera image processing system in one embodiment.

[0045] Figure 8 for Figure 7 Workflow diagram of an industrial camera image processing system. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0048] For ease of understanding, some terms used in the embodiments of this application are explained as follows:

[0049] ISP (Image Signal Processor);

[0050] VI (Video Input);

[0051] VPSS (Video Process Sub-System).

[0052] VENC (Video Encoder);

[0053] I2C (Inter-integrated Circuit);

[0054] VPU (Video Processing Unit);

[0055] TPU (Tensor Processing Unit).

[0056] In one embodiment, Figure 2A structural block diagram of an industrial camera image processing system is provided, including a video encoding module 1 and an industrial camera 2, which are connected. The industrial camera 2 includes an image acquisition module 21 and an algorithm recognition module 22. The first output port of the image acquisition module 21 is connected to the input port of the video encoding module 1 to form a first transmission path, and the second output port of the image acquisition module 21 is connected to the input port of the algorithm recognition module 22 to form a second transmission path.

[0057] The algorithm recognition module 22 may include one or more ( Figure 2 (Only one processor is shown in the image.) The processor can be a microprocessor (MCU) or a programmable logic device (FPGA), etc.

[0058] In some embodiments, the industrial camera 2 may further include a transmission module for communication functions, used to receive or send data via a network. This network includes a wireless network provided by the industrial camera's communication vendor. The transmission module may include a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. The transmission module may also be a radio frequency (RF) module, used for wireless communication with the Internet.

[0059] The industrial camera image processing system adopts a multi-threaded parallel processing architecture, including the following threads:

[0060] Image acquisition thread (priority SCHED_FIFO 20): std::thread grab_thread, used to perform image acquisition;

[0061] OSD overlay thread: std::thread osd_thread, used to overlay timestamps or custom text onto images;

[0062] The video encoding thread, std::thread venc_thread, is used to encode images.

[0063] RGB push thread (priority SCHED_FIFO 20): std::thread push_rgb_thread, used to convert image formats to RGB format.

[0064] Figure 3 This is a flowchart illustrating the industrial camera image processing method in this embodiment, which includes the following steps:

[0065] In step S101, the image acquisition module 21 acquires images of the surrounding environment and generates an initial image.

[0066] The industrial camera 2 can be one or more, including an RGB camera and / or an infrared camera. The image acquisition module 21 can be an RGB sensor or an infrared sensor. The industrial camera 2 can be controlled by a V4L2 (Video4Linux2, a Linux video device driver framework), accessing the RGB camera via a MIPI interface and the infrared camera via a USB interface.

[0067] During the initial image acquisition process, a camera device object can be created first, configuring different device paths for different camera modes (RGB / infrared). For example, the RGB camera path is / dev / video0 (MIPI interface), and the infrared camera path is / dev / video6 (USB interface). The open_v4l2reader function is then called to initialize the V4L2 device, control the industrial camera 2 to start, activate the image acquisition thread, and output streaming data. The streaming data contains the initial image, which is an NV21 frame generated from RAW data after format conversion. NV21 is a YUV image format.

[0068] In step S102, the video encoding module 1 obtains the initial image through the first transmission path, encodes the initial image, and generates an encoded video stream.

[0069] Video encoding module 1 executes the OSD overlay thread and the video encoding thread. First, it performs OSD overlay and scaling on the initial image (NV21 frames), and then performs video encoding. OSD overlay refers to overlaying a timestamp (displaying the current time) or custom text onto the initial image. During video encoding, the H.264 encoding standard is used to compress the video stream, and the encoded video stream is transmitted via the RTSP protocol. The final output is sent to a web client or other clients for remote viewing or monitoring.

[0070] In step S103, the algorithm recognition module 22 acquires the initial image through the second transmission path, performs algorithm recognition based on the initial image, and generates the algorithm recognition result.

[0071] The algorithm recognition module 22 executes the RGB push thread to perform format conversion and resolution adjustment on the initial image. Internally, the algorithm recognition module 22 includes a hardware accelerator that converts the input NV21 frame from YUV format to RGB format, obtaining an RGB frame. In RGB format, each pixel occupies 3 bytes and is arranged in blue-green-red order. The hardware accelerator can be a VPSS (Video Process Sub-System). After obtaining the RGB frame, algorithm recognition is performed. For example, model inference can be used to perform target detection and tracking on the RGB frame, generating signals based on the detection and tracking results to control the operating state of mechanical equipment, such as starting or stopping. These signals can be DO (Digital Output) signals.

[0072] Traditional industrial camera image processing systems involve multiple conversions from the camera's original format to the encoding / decoding format and then to the format required by the algorithm. This results in a long signal output delay as the image data passes through multiple intermediate stages in the processing chain. In steps S101 to S103, the image acquisition module 21 and the algorithm recognition module 22 are integrated into the industrial camera 2. The algorithm recognition module 22 and the video encoding module 1 are connected to the image acquisition module 21, forming two transmission paths. This allows video encoding and algorithm recognition to be processed in parallel. When the image data reaches the algorithm recognition module 22, it does not need to undergo multiple format conversions and encoding / decoding processes, reducing signal output delay and improving the response speed of the industrial camera image processing system.

[0073] In one embodiment, Figure 4 Another structural block diagram of an industrial camera image processing system is provided, such as Figure 4 As shown, the algorithm recognition module 22 includes a preprocessing unit 221, a buffer management unit 222, and an algorithm unit 223 connected in sequence. Figure 5 The internal operation flowchart of the algorithm recognition module 22 is provided, such as Figure 5 As shown, in step S103, the algorithm recognition module performs algorithm recognition based on the initial image and generates the algorithm recognition result, which can be achieved through the following method:

[0074] In step S201, the preprocessing unit 221 acquires the initial image in zero-copy mode, preprocesses the initial image, and generates a preprocessed image.

[0075] The preprocessing unit 221 performs scaling and / or format conversion on the initial image to obtain a preprocessed image. For example, the preprocessing unit 221 performs format conversion and resolution adjustment on the initial image. The preprocessing unit 221 may employ a hardware accelerator to convert the input NV21 frame from YUV format to RGB format, obtaining an RGB frame. In the RGB format, each pixel occupies 3 bytes and is arranged in blue-green-red order. The hardware accelerator may be a VPSS (Video Process Sub-System).

[0076] Zero-copy mode refers to directly mapping the kernel buffer to user space through mmap (memory-mapped files), avoiding data copying from kernel mode to user mode and reducing data transfer overhead.

[0077] In step S202, the buffer management unit 222 buffers the preprocessed image.

[0078] The buffer management unit 222 receives the preprocessed image and buffers it into the corresponding queues. The queues include an RGB frame push queue, an OSD processing queue, and a scaling processing queue. The RGB frame push queue is a queue of image frames that have been converted to RGB format, the OSD processing queue is a queue of image frames with overlaid characters (timestamps or custom text), and the scaling processing queue is a queue of image frames that have been scaled.

[0079] In some embodiments, Figure 6 A flowchart of queue management for a buffer management unit 222 is provided, such as... Figure 6 As shown, after the buffer management unit 222 receives the preprocessed image and buffers the preprocessed image into the corresponding queue, the method further includes the following steps:

[0080] Step S301: Acquire the mutex lock. A mutex lock is used to protect access to shared resources; only one thread can access the shared resource at a time.

[0081] Step S302: Monitor the current capacity of each queue.

[0082] Step S303: When it is detected that the current capacity of any queue exceeds its corresponding capacity limit, the oldest frame in that queue is discarded.

[0083] Different queues employ different capacity limits, and the cache depth is optimized based on processing characteristics. The OSD processing queue capacity limit can be set relatively small (e.g., 3 frames) to ensure low latency, while the RGB frame push queue capacity limit is moderate (e.g., 10 frames) to balance throughput and latency. When it is detected that the current capacity of any queue exceeds its corresponding capacity limit, the oldest frame in that queue can be discarded by removing the head element.

[0084] Step S304: Determine whether the current capacity of the queue exceeds N times its capacity limit.

[0085] Step S305: If the current capacity of the queue exceeds N times its capacity limit, output a warning log to record that the queue is severely congested.

[0086] N can be an integer greater than or equal to 1. For example, when the current capacity of the queue exceeds three times its capacity limit, a warning log is output to indicate that the queue is severely congested.

[0087] Step S306: Release the mutex lock.

[0088] When a thread attempts to acquire a mutex, if the mutex is already held by another thread, the current thread blocks and waits. Once the mutex is successfully acquired, execution enters the critical section; that is, only the thread holding the mutex can execute the critical section code, and other threads must wait for the mutex to be released. The mutex is automatically released when its scope ends to avoid deadlocks caused by forgetting to release it. This embodiment, by introducing a mutex, ensures that only one thread can access shared resources at a time, preventing data races and inconsistent states.

[0089] In step S203, the algorithm unit 223 reads the preprocessed image from the buffer management unit, performs algorithm recognition on the preprocessed image, and generates the algorithm recognition result.

[0090] Algorithm unit 223 reads RGB frames from the RGB frame push queue of buffer management unit 222; algorithm unit 223 checks whether a user callback function has been registered; if a user callback function has been registered in algorithm unit 223, the user callback function is called to pass the RGB frame; the return value of the user callback function is obtained, and the algorithm recognition result is obtained based on the return value.

[0091] In this step, algorithm unit 223 performs algorithmic recognition on the RGB frames, such as using model inference to perform target detection and tracking on the RGB frames, and generates a signal (i.e., algorithm recognition result) based on the target detection and tracking results to control the operating state of the mechanical equipment, such as starting or stopping. The signal can be a DO (Digital Output) signal.

[0092] This embodiment uses a buffer management unit 222 in the algorithm recognition module 22 to cache each queue, replacing the traditional switch and reducing hardware costs. Furthermore, the buffer management unit 222 has adaptive queue management capabilities, improving the throughput performance of the algorithm recognition module 22.

[0093] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0094] In one embodiment, Figure 7 A schematic diagram of the architecture of an industrial camera image processing system is provided, such as... Figure 7 As shown, the architecture of this industrial camera image processing system includes: an external connection layer, a hardware layer, a driver layer, and a business layer.

[0095] External connection layer: This includes industrial cameras and a PLC (Programmable Logic Controller). Sensor1 and Sensor2 represent two types of industrial cameras (RGB camera and infrared camera), responsible for acquiring images. The PLC receives signals output by the system (e.g., DO signals) to control the mechanical equipment.

[0096] Hardware Layer: This includes hardware acceleration resources and general peripherals. Hardware acceleration resources include: VPU (Video Processing Unit), dedicated to video decoding / encoding to improve video stream processing efficiency; TPU (Tensor Processing Unit), providing high-performance computing power for AI model inference and accelerating recognition speed; VI / ISP (Video Input / Image Signal Processor), responsible for receiving and preprocessing image sensor data (such as noise reduction and white balance); VENC (Video Encoder), used to complete video compression encoding tasks, supporting formats such as H.264 / H.265. General peripherals include: I2C, used to configure sensor parameters (such as exposure time and gain); MIPI, a high-speed serial interface for transmitting RAW image data from the sensor; USB, used for firmware upgrades or debugging communication; and GPIO (General Purpose Input / Output), general-purpose digital I / O, used for trigger signals or status indicator control.

[0097] Driver layer: This includes the Kernel and SDK (Software Development Kit). The Kernel is the core of the operating system, managing hardware resource scheduling and process execution. The SDK provides standardized API interfaces, encapsulating low-level hardware operations for upper-layer applications to call.

[0098] Business layer: The main functions are as follows:

[0099] API: An interface service exposed to the outside world for use by host computers or web applications.

[0100] Thread management: Coordinates the execution order and resource allocation of multiple concurrent tasks (such as image acquisition, encoding, and inference).

[0101] Queue Management: Use buffered queues to manage the flow of image frames and prevent data backlog or loss.

[0102] Image processing: Preprocessing operations such as format conversion, scaling, and filtering are performed on the initial image.

[0103] Video encoding: Encodes the processed image into a standard video stream (such as RTSP) for remote monitoring.

[0104] Camera management: Configure and control the operating parameters (such as resolution and frame rate) of multiple cameras.

[0105] Control output: Generate DO signals based on the algorithm results and send them to the PLC to achieve automatic control.

[0106] Model inference: Calling pre-trained AI models (such as object detection and classification) to perform intelligent analysis on images.

[0107] Business Management: Responsible for global scheduling and policy control, such as starting / stopping an industrial camera, setting algorithm recognition rules for different scenarios, monitoring system operation status, and uniformly managing the deployment and updates of multiple devices.

[0108] In this embodiment, Figure 8 A flowchart of the industrial camera image processing system is provided, such as... Figure 8 As shown, in the industrial camera 2, the image acquisition module 21 is configured via the I2C interface. After acquiring images of the surrounding environment, it transmits the acquired RAW data via the MIPI interface, then converts the RAW format to YUV format to generate NV21 frames, which is the initial image. The initial image is sent to the video encoding module 1 via the first transmission path and to the algorithm recognition module 22 via the second transmission path.

[0109] In video encoding module 1, the OSD overlay thread and video encoding thread are executed. First, OSD overlay and scaling are performed on the initial image (NV21 frames), followed by video encoding. OSD overlay refers to overlaying a timestamp (displaying the current time) or custom text onto the initial image. During video encoding, the H.264 encoding standard is used to compress the video stream, and the encoded video stream is transmitted via the RTSP protocol. Finally, it is output to a web interface via the network for remote viewing or monitoring.

[0110] In the algorithm recognition module 22, an RGB push thread is executed to perform format conversion and resolution adjustment on the initial image. The algorithm recognition module 22 internally includes a hardware accelerator that converts the input NV21 frame from YUV format to RGB format, obtaining an RGB frame. In RGB format, each pixel occupies 3 bytes and is arranged in blue-green-red order. The hardware accelerator can be a VPSS (Video Process Sub-System). After obtaining the RGB frame, algorithm recognition is performed. For example, model inference can be used to perform target detection and tracking on the RGB frame, and signals can be generated based on the detection and tracking results to control the operating state of mechanical equipment, such as starting or stopping. These signals can be DO (Digital Output) signals.

[0111] Specifically, the algorithm recognition module 22 includes a preprocessing unit 221, a buffer management unit 222, and an algorithm unit 223 connected in sequence. The preprocessing unit 221 acquires the initial image in zero-copy mode, preprocesses the initial image, and generates a preprocessed image. The buffer management unit 222 buffers the preprocessed image. The algorithm unit 223 reads the preprocessed image from the buffer management unit, performs algorithm recognition on the preprocessed image, and generates an algorithm recognition result. The specific processing steps of each unit can be found in the above embodiment description, and will not be repeated here.

[0112] In this embodiment, the image acquisition module 21 and the algorithm recognition module 22 are integrated into the industrial camera 2. The algorithm recognition module 22 and the video encoding module 1 are respectively connected to the image acquisition module 21, forming two transmission paths. This allows video encoding and algorithm recognition to be processed in parallel. When image data reaches the algorithm recognition module 22, it does not need to undergo multiple format conversions and encoding / decoding processes, reducing signal output latency and improving the response speed of the industrial camera image processing system. Furthermore, by setting a buffer management unit 222 in the algorithm recognition module 22 to buffer each queue, a traditional switch is replaced, reducing hardware costs. The buffer management unit 222 also has adaptive queue management capabilities, improving the throughput performance of the algorithm recognition module 22.

[0113] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above method embodiments.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0115] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method for an industrial camera, characterized in that, An industrial camera image processing system is applied, comprising an industrial camera and a video encoding module. The industrial camera internally includes an image acquisition module and an algorithm recognition module. A first output port of the image acquisition module is connected to the input port of the video encoding module to form a first transmission path, and a second output port of the image acquisition module is connected to the input port of the algorithm recognition module to form a second transmission path. The method includes: The image acquisition module acquires images of the surrounding environment and generates an initial image; The video encoding module acquires the initial image through the first transmission path, encodes the initial image, and generates an encoded video stream. The algorithm recognition module acquires the initial image through the second transmission path, performs algorithm recognition based on the initial image, and generates algorithm recognition results.

2. The industrial camera image processing method according to claim 1, characterized in that, The algorithm recognition module includes a preprocessing unit, a cache management unit, and an algorithm unit connected in sequence. The algorithm recognition module performs algorithm recognition based on the initial image and generates algorithm recognition results, including: The preprocessing unit acquires the initial image in zero-copy mode, preprocesses the initial image, and generates a preprocessed image. The cache management unit caches the preprocessed image; The algorithm unit reads the preprocessed image from the cache management unit, performs algorithm recognition on the preprocessed image, and generates the algorithm recognition result.

3. The industrial camera image processing method according to claim 2, characterized in that, The preprocessing unit preprocesses the initial image to generate a preprocessed image, including: The preprocessing unit performs scaling and / or format conversion on the initial image to obtain the preprocessed image.

4. The industrial camera image processing method according to claim 3, characterized in that, The buffer management unit caches the preprocessed image, including: The buffer management unit receives the preprocessed image and caches the preprocessed image into the corresponding queue, wherein the queue includes an RGB frame push queue, an OSD processing queue, and a scaling processing queue.

5. The industrial camera image processing method according to claim 4, characterized in that, After the buffer management unit receives the preprocessed image and caches the preprocessed image into the corresponding queue, the method further includes: Acquire the mutex lock; Monitor the current capacity of each queue; When it is detected that the current capacity of any queue exceeds its corresponding capacity limit, the oldest frame in that queue is discarded. Release the mutex.

6. The industrial camera image processing method according to claim 5, characterized in that, When it is detected that the current capacity of any queue exceeds its corresponding capacity limit, the method further includes: Determine whether the current capacity of the queue exceeds its capacity limit by N times; If the current capacity of the queue exceeds N times its capacity limit, a warning log will be output to record that the queue is severely congested.

7. The industrial camera image processing method according to claim 4, characterized in that, The algorithm unit reads the preprocessed image from the cache management unit, performs algorithm recognition on the preprocessed image, and generates the algorithm recognition result, including: The algorithm unit reads RGB frames from the RGB frame push queue of the buffer management unit; The algorithm unit checks whether a user callback function has been registered; If the user callback function has been registered in the algorithm unit, then the user callback function is invoked to pass the RGB frame; Obtain the return value of the user callback function, and obtain the algorithm recognition result based on the return value.

8. An industrial camera, characterized in that, include: The system includes an image acquisition module and an algorithm recognition module. The first output port of the image acquisition module is connected to the input port of the video encoding module to form a first transmission path, and the second output port of the image acquisition module is connected to the input port of the algorithm recognition module to form a second transmission path. The image acquisition module is used to acquire images of the surrounding environment and generate an initial image; The video encoding module is used to acquire the initial image through the first transmission path, encode the initial image, and generate an encoded video stream. The algorithm recognition module is used to acquire the initial image through the second transmission path, and perform algorithm recognition based on the initial image to generate algorithm recognition results.

9. An industrial camera image processing system, characterized in that, include: The video encoding module and the industrial camera of claim 8 are connected.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images