Application layer flow control method based on image network transmission

Abstract

The invention discloses an application layer flow control method based on image network transmission, and relates to the technical field of data transmission and management.The method comprises the steps that a camera SDK is used for creating an acquisition queue containing a buffer in a host application layer, and the host application layer can receive image data sent by a camera through the acquisition queue; the method comprises the following steps of: receiving a buffer, putting the corresponding buffer into a processing queue to inform a user to process, feeding back the condition of the buffer to a camera in real time by a host application layer through a communication link in the process, determining whether to continuously send an image or not by the camera according to a feedback result, and stopping image transmission in time when the buffer capability of a host is insufficient to avoid data loss; in the technical scheme of the invention, a host application layer can feed back the real-time information of the buffer in the acquisition queue to the camera through a link, and when the host does not have a buffer area, the transmission is suspended and the image is stored in the buffer of the camera, so that the flow control from a data source to the application layer is realized; the problem of transmission stability in network industrial camera application is effectively solved, and the network transmission reliability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission and management, and in particular to an application layer flow control method based on image network transmission. Background Art

[0002] Industrial cameras with network interfaces have a high market share. Currently, industrial network interfaces support multiple rate standards such as 1GigE, 2.5GigE, 5GigE and 10GigE. Among them, industrial cameras with 10GigE interfaces are considered to be the most cost-effective high-bandwidth industrial cameras compared with industrial cameras with interfaces such as CamerLink, CXP, and HSLink, due to their low cable cost, long transmission distance and the ability to use universal network cards. However, the cameras do not use DMA transmission, and there is unstable image transmission in industrial image acquisition systems. One of the reasons is physical damage or deformation caused by interference or insufficient driving capability of the data signal during transmission, and the other is data loss or overwriting caused by buffer overflow in the transmission path. The former can be avoided by improving the anti-interference ability of the system and cables and enhancing the driving capability of the signal, while the latter needs to be solved by using the link buffer in conjunction with the data back pressure mechanism.

[0003] The back pressure mechanism is to transmit pressure layer by layer, and the pressure here is the early warning of insufficient buffering capacity. When the buffering capacity of a certain subsystem is about to reach saturation, it is necessary to inform the previous subsystem. Therefore, in the design of the acquisition system, the continuity and perfection of the back pressure mechanism are very important. However, in the industrial acquisition system, this continuity cannot be well guaranteed. Industrial cameras based on network interfaces have such problems. The discontinuity of the data acquisition channel buffer overflow feedback mechanism will cause the system to be unable to guarantee stable image transmission under high bandwidth and high CPU occupancy.

[0004] Traditional network interface industrial cameras are based on UDP protocol for transmission, and usually use Pause frames for flow control at the link layer. When the buffer of the network card or switch is about to overflow, the network card or switch will send a Pause frame to notify the camera. After the camera receives the Pause frame, it will pause data transmission and temporarily store the image data in the buffer inside the camera. After the pause ends, the image data transmission will be resumed. The Pause frame method can well solve the problem of instantaneous blocking of switches or network cards in multi-camera mode, but it is far from enough for the entire acquisition chain.

[0005] The data path of the network interface industrial camera is as follows Figure 7As shown, data is generated from the image sensor on the camera side, flows through the logic unit inside the FPGA (Field-Programmable Gate Array) to enter the network MAC (Medium Access Control), the network physical layer, enters the host side network card through the network cable, and is then submitted to the user program through the network card driver layer and the filter driver layer. Among them, the filter driver is used to implement parsing and filtering of network image data packets in a specific format. It cannot use a blocking method to copy buffer data between it and the network card driver. The reason is that when blocking occurs (for example, there is no buffer available in the upper layer, or the CPU is cut out because of executing a higher priority thread or interrupt service), if the filter driver does not return the buffer of the network card driver in time, the network card will not be able to receive any network packets. Since the network is a public resource, the network card must continue to receive network data packets, so this situation is not allowed. Usually, a non-blocking method is used between the network card and the driver, but when there is no buffer available in the upper layer of the network card driver, the network card driver cannot transmit data to the filter driver, which will cause the network card driver's buffer to be full and unable to release it in time, and unable to continue to receive new data packets, resulting in data loss. After the buffer of the network card driver is full, it cannot receive the feedback signal of the filter driver. Therefore, in the entire acquisition chain, the feedback will be interrupted between the filter driver and the network card driver, resulting in the inability of the back pressure mechanism to be continuous, and then there is a risk of buffer overflow, resulting in unreliable data transmission. Summary of the invention

[0006] The purpose of the present invention is to propose a flow control mechanism based on the application layer to solve the problem of feedback interruption between the filter driver and the network card driver in the industrial camera acquisition chain, and directly establish a feedback mechanism between the application side and the data source end to achieve the purpose of flow control. In this way, the problem of discontinuity of the feedback mechanism in the industrial image acquisition chain is solved, so that when the data is blocked at the application layer, the blocking information can be fed back to the industrial camera in time, and the buffer inside the camera can be fully utilized to eliminate the impact of instantaneous data blocking, thereby improving the stability of the network interface industrial image acquisition system and the reliability of image data transmission.

[0007] The technical solution of the present invention is to provide an application layer flow control method based on image network transmission, which is applied to the process of industrial camera transmitting image data to host through network interface. The process adopts layer-by-layer back pressure mechanism for data transmission flow control. The host side includes network card driver and filter driver. The network card driver is used to transmit the received image data to the filter driver. The method includes:

[0008] Step 1: Use the camera SDK to allocate a group of buffers at the host application layer, and create a corresponding acquisition queue for receiving image data from the industrial camera. Add the buffer to the acquisition queue as a data receiving unit, and map the buffer to the host's filter driver layer, where the mapping is to pass the buffer's descriptor to the filter driver layer, so that the filter driver layer can transfer the image to the acquisition queue;

[0009] Step 2: The host application layer establishes direct communication with the camera via the network card, sends the cumulative mapping buffer number to the camera, and the camera starts to collect images and sends them to the host. The host application layer receives the images through the collection queue and puts the buffer into the processing queue to notify the user to process;

[0010] Step 3: The host application layer puts the processed buffer back into the acquisition queue and maps it to the filter driver layer;

[0011] Step 4: The host application layer sends the cumulative number of mapping buffers to the camera. The camera checks the current cumulative number of mapping buffers and the number of images sent. If the current cumulative number of mapping buffers is greater than the number of images sent, execute step 5. If the current cumulative number of mapping buffers is less than or equal to the number of images sent, execute step 6.

[0012] Step 5: The camera continues to send images to the host, and the host application layer continues to receive images through the acquisition queue and puts the buffer into the processing queue to notify the user to process. After the processing is completed, it returns to step 3.

[0013] Step 6: The camera stops sending images to the host and caches the currently acquired image data into the camera's buffer. When the current cumulative number of mapped buffers is greater than the number of images sent, the camera returns to step 5.

[0014] Further, the register is set in the host, and step 2 specifically includes:

[0015] The host application layer obtains the cumulative number of mapped buffers mapped to the filter driver layer through the register and sends a notification to the camera. The camera starts to collect images and sends them frame by frame to the host. The images enter the host-side network card through the network cable, and then pass through the network card driver layer and the filter driver layer in turn, and finally enter the host application layer. The host application layer receives the image through the acquisition queue and stores it in the corresponding buffer, and then puts the buffer in the acquisition queue that has completed image storage into the processing queue and notifies the user to process it.

[0016] Furthermore, the buffer processed in step 3 is released, the host application layer marks the released buffer as available, puts it back into the acquisition queue, and maps it to the filter driver layer again, and the register updates the accumulated number of mapped buffers.

[0017] Furthermore, step 4 specifically includes:

[0018] Step 4.1, the host application layer determines the frequency of sending notifications of the current cumulative number of mapped buffers to the camera based on the total number of allocated buffers, wherein the frequency of sending notifications decreases as the total number of buffers increases;

[0019] Step 4.2, the host application layer sends a notification of the current cumulative number of mapping buffers to the camera at a frequency, and starts a timer synchronously after the sending is completed. If the cumulative number of mapping buffers remains unchanged within the preset time, the host application layer sends a notification of the current cumulative number of mapping buffers to the camera again. If the cumulative number of mapping buffers increases within the preset time, the timer is reset after the next notification;

[0020] In step 4.3, the camera extracts the notification of the cumulative number of mapped buffers through the FPGA module, obtains the number of images sent, and compares the current cumulative number of mapped buffers with the number of images sent. If the current cumulative number of mapped buffers is greater than the number of images sent, it is determined that the host application layer has available buffers. If the current cumulative number of mapped buffers is less than or equal to the number of images sent, it is determined that the host application layer has no available buffers.

[0021] Furthermore, the relationship between frequency and the total number of buffers is:

[0022]

[0023] Where f is the frequency at which the host application layer sends notifications to the camera, the unit is the number of times the buffer is mapped to the filter driver layer, and N is the total number of allocated buffers.

[0024] Furthermore, in step 4.2, when the timer reaches the preset value and the cumulative number of mapping buffers remains unchanged, the host application layer determines that multiple consecutive notifications have not reached the camera, and the industrial image acquisition system enters a deadlock state, or determines that the user takes too long to process the image, and the host application layer actively sends a notification of the cumulative number of mapping buffers to the camera, wherein the cumulative number of mapping buffers sent is the same as the previous time.

[0025] Furthermore, in step 6, the camera caches the currently acquired image data into the camera's buffer and waits for the next notification from the host. If after the next notification, the cumulative number of mapped buffers is greater than the number of images sent, the camera determines that the host application layer has available buffers and returns to execute step 5. If after the next notification, the cumulative number of mapped buffers is less than or equal to the number of images sent, the camera determines that the host application layer has no available buffers and continues to wait for notification.

[0026] Furthermore, step 1 specifically includes:

[0027] Import the camera SDK in the host application layer, obtain the camera's device information through the interface API provided by the camera SDK, establish a communication link with the camera based on the device information, use the camera SDK to allocate a set of buffers for storing images in the memory of the host application layer, and then define an acquisition queue for receiving images. Pass all buffer descriptors to the acquisition queue so that the acquisition queue can recognize the buffer and store the image data in the buffer after receiving it. Finally, map the buffer in the acquisition queue to the host's driver layer so that the filter driver layer can smoothly transfer the image to the acquisition queue.

[0028] The beneficial effects of the present invention are:

[0029] First, the technical solution of the present invention is to design an application layer flow control method to solve the problem of discontinuous feedback mechanism of data acquisition path buffer overflow in network industrial camera applications. The method uses the camera SDK to establish a communication link between the host application layer and the camera. The host application layer can feed back the real-time information of the buffer in the acquisition queue to the camera through the communication link. When the host has no buffer for data reception, the camera will suspend image transmission and write the image into the camera's buffer area, thereby realizing flow control from the data source to the application layer.

[0030] In the prior art, the host application layer usually adopts a back pressure mechanism to feedback the pressure warning of insufficient host buffer capacity to the camera layer by layer. When the network card driver buffer is full and the filter driver fails to return the buffer in time, the network card driver can no longer receive the image data sent by the camera, nor can it receive the pressure warning of the filter driver. At this time, the back pressure mechanism is no longer continuous, and the camera side cannot know that the host buffer capacity is insufficient, so it will continue to send image data, and the network card driver cannot continue to receive image data, resulting in data loss. The technical solution in the present invention utilizes the communication link between the host application layer and the camera to feedback the host buffer capacity insufficient to the camera, avoiding the problem of discontinuous back pressure mechanism in the prior art, so that the camera can stop sending images in time when the host buffer capacity is insufficient, avoiding data loss, and also making full use of the camera's buffer.

[0031] The technical solution of the present invention effectively solves the problem of transmission stability of network cameras, especially 10GigE network cameras. While maintaining the low cost and transmission distance advantages of network interface industrial cameras, it improves network transmission reliability and enhances the competitiveness of interface industrial cameras.

[0032] Second, in the preferred implementation of the present invention, a non-response design is used in the communication of the host to the camera feedback buffer, thereby avoiding the problem of data loss or delayed notification due to a large amount of uplink bandwidth being occupied, reducing the risk of notification failure, and avoiding retransmission storms caused by high frame rate, response loss or timeout; at the same time, the notification content sent by the host is set to the cumulative number of mapped buffers, so as to solve the problem that the camera cannot determine how many available buffers exist in the host due to a certain notification failure. In this way, even if a certain notification cannot reach the camera, as long as a subsequent communication can successfully reach the camera and update the data, the synchronization between the camera and the host will not be affected, and regular error correction synchronization between the host and the camera can be achieved.

[0033] Third, in the preferred implementation of the present invention, a timed sending mechanism is designed on the host side for the situation where multiple consecutive notifications fail to reach the camera correctly. The timer is started synchronously after each notification is sent. If no processed buffer is received within the specified time, it is determined that multiple consecutive notifications have failed to reach the camera correctly or the user takes too long to process the image. The host application layer actively sends a notification of the cumulative number of mapping buffers to the camera to avoid the industrial image acquisition system entering a deadlock state and interrupting operation.

[0034] Fourth, in the preferred implementation of the present invention, the frequency of the host application layer sending notifications of the cumulative number of mapped buffers to the camera is adjusted according to the total number of allocated buffers to avoid the problem of too high notification interaction frequency caused by the industrial camera working in small window mode; at the same time, FPGA is used on the camera side to filter notifications of the cumulative number of mapped buffers to avoid firmware interrupts in the camera after the notification reaches the camera firmware, thereby ensuring the stable operation of the camera firmware. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The advantages of the above and / or additional aspects of the present invention will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0036] Figure 1 is a schematic diagram of the structure of an industrial image acquisition system according to an embodiment of the present invention;

[0037] Figure 2is a schematic flow chart of an application layer flow control method based on image network transmission according to an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of operation when a response design is adopted between a host and a camera according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of comparing the cumulative number of mapping buffers and the number of sent images according to an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of a camera filtering notifications through an FPGA according to an embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of the operation of a host-side timing sending mechanism according to an embodiment of the present invention;

[0042] Figure 7 It is a schematic diagram of the host feeding back pressure to the camera layer by layer when the back pressure mechanism is adopted in the prior art. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0045] Industrial image acquisition systems such as Figure 7 As shown in the figure, the camera side includes an image sensor, FPGA and network physical layer, where FPGA includes a logic unit and a network MAC; the host side includes a network card, a network card driver layer, a filter driver layer and a host application layer, where the network card driver layer and the filter driver layer belong to the host driver layer. Image data is generated from the image sensor on the camera side, flows through the logic unit and network MAC inside the FPGA, enters the network physical layer, and then enters the host side network card through the network cable, and is submitted to the user program after passing through the network card driver layer and the filter driver layer.

[0046] like Figures 1 to 6As shown, this embodiment provides an application layer flow control method based on image network transmission. The method is applied to the process of industrial camera transmitting image data to the host through the network interface. The process adopts the back pressure mechanism transmitted layer by layer to control the data transmission flow. The host side includes a network card driver and a filter driver. The network card driver is used to transmit the image data received to the filter driver. The application layer flow control based on image network transmission includes: bypassing the driver

[0047] Step 1, the host application layer establishes a link with the camera through the camera SDK, uses the camera SDK to allocate a predetermined number of buffers in the host application layer, and creates a corresponding acquisition queue for receiving image data from the industrial camera, adds the buffer to the acquisition queue as a data receiving unit, and maps the buffer in the acquisition queue to the filter driver layer of the host, wherein the mapping is to pass the descriptor of the buffer to the filter driver layer, so that the filter driver layer can transmit the image to the acquisition queue;

[0048] In this embodiment, the camera SDK (Software Development Kit) provides a set of application programming interfaces (APIs), which enable the host application layer to communicate and interact with the camera hardware through the interface APIs, thereby realizing the management and control of the camera by the host application layer.

[0049] Specifically, the camera SDK is imported and initialized at the host application layer. After the initialization is completed, the camera device information is obtained through the interface API provided by the camera SDK, and a communication link is established with the camera according to the camera device information. After the communication link is established, the various parameters of the camera can be configured through the interface API of the camera SDK (the API allows the host application layer to adjust various parameters of the camera), such as resolution, frame rate, etc.;

[0050] According to the actual acquisition needs, the user uses the interface of the camera SDK to allocate a group of buffers in the memory of the host application layer, where each buffer will be used to store a frame of image; then use the interface of the camera SDK to define a collection queue of a predetermined size in the host application layer that can receive image data in an orderly manner, where the predetermined size refers to the number of buffers that can be managed. The allocated buffers will not be automatically added to the acquisition queue, and they need to be manually registered to the queue, that is, the buffer descriptor is passed to the acquisition queue so that the acquisition queue can recognize the buffer and store the image data in the buffer after receiving the image data, that is, create an acquisition queue and add all buffers to the acquisition queue, where the acquisition queue is used to manage the allocated buffers. After acquiring the image, it can write the input image data into the corresponding buffer according to certain rules and order; the buffer descriptor includes memory address and metadata. The memory address includes the starting address (the starting address of the buffer in the host memory, used to indicate the location of data storage), buffer size (indicating the capacity of the buffer, that is, how many bytes of data can be stored), and memory alignment information (indicating the arrangement of the buffer in the memory to ensure that data can be read and written efficiently); metadata includes image format, resolution, bit depth per pixel (describing the number of bits used for each pixel, such as 8 bits, 10 bits or 16 bits), row alignment (the number of bytes of each row of image data in the buffer to ensure memory alignment performance), frame ID (frame sequence number, used to distinguish different frames in the data stream), etc.

[0051] Map the buffer in the acquisition queue to the filter driver layer to ensure that the filter driver can smoothly transfer the processed image data to the acquisition queue of the host application layer.

[0052] Step 2: The host application layer establishes direct communication with the camera via the network card and sends the accumulated mapping buffer quantity to the camera. The camera starts to collect images and sends them to the host. The host application layer receives the images through the collection queue and puts the buffer into the processing queue to notify the user to process.

[0053] Before the camera starts to capture images, the host initializes registers and uses registers to record the cumulative number of buffers mapped to the filter driver layer. The host application layer obtains the cumulative number of mapped buffers mapped to the filter driver layer through registers, and sends a notification of the current cumulative number of mapped buffers to the camera, informing the camera of the cumulative number of mapped buffers. The camera starts to capture images, and after determining that there are available buffers in the host, it sends images to the host frame by frame. The images enter the host side network card through the network cable, and then pass through the network card driver layer and the filter driver layer in turn, and finally enter the host application layer. The host application layer receives the image through the acquisition queue and stores the image in the corresponding buffer, and then puts the buffer that has completed image storage in the acquisition queue into the processing queue, and notifies the user to process it through callback (that is, the host application layer will automatically put the buffer into the processing queue after a buffer in the acquisition queue completes image storage, and notify the user to process it through the callback function, wherein the callback function is triggered when the buffer in the acquisition queue completes image storage).

[0054] In this embodiment, when the network card driver buffer is full, the pressure information fed back by the filter driver cannot be passed to the camera side, and image data cannot be received any more, which will cause the image data to be lost. In the host application layer, direct communication with the camera via the network card is established, and the host application layer sends the accumulated mapping buffer quantity to the camera directly through the network card, so that the pressure information feedback on the host side skips the network card driver and the filter driver (in this process, the network card driver still transmits the received image data to the filter driver), so that the pressure information can be passed to the camera in time after the host side buffer is full, avoiding the problem of image data loss caused by the inability to pass the pressure information fed back by the filter driver to the camera side after the network card driver buffer is full.

[0055] Step 3: The host application layer puts the processed buffer back into the acquisition queue and maps it to the host's filter driver layer.

[0056] After the user completes the processing, the buffer is released. The host application layer will mark the released buffer as available, put it back into the acquisition queue, and map it to the filter driver layer again, waiting to store the image again. At the same time, the host register updates the cumulative number of mapped buffers.

[0057] Step 4: The host application layer sends the current cumulative number of mapped buffers to the camera. The camera checks the current cumulative number of mapped buffers and the number of images sent. Based on the comparison result of the current cumulative number of mapped buffers and the number of images sent, it determines whether the host application layer has available buffers. If the current cumulative number of mapped buffers is greater than the number of images sent, it is determined that there is available buffer and step 5 is executed. If the current cumulative number of mapped buffers is less than or equal to the number of images sent, it is determined that there is no available buffer and step 6 is executed.

[0058] In this embodiment, the number of cumulative mapping buffers continuously updated in the host application layer is used as the basis for judging whether the camera should send images to the host. Therefore, reliable communication between the host and the camera is very important. The communication includes a response design and a no-response design. In order to ensure that the notification sent by the host can correctly reach the camera, this application preferably uses a no-response design between the host and the camera, such as Figure 3 As shown, the no-response design means that the camera does not need to confirm with the host each time it receives a notification. If there is no guarantee that every notification can reach the camera, the content of the notification needs to be adjusted to facilitate regular error correction and synchronization between the host and camera sides.

[0059] The notification content sent by the host is set to the cumulative number of mapped buffers. This requires setting a register on the host side to record the cumulative number of buffers mapped to the filter driver layer. That is, each time the host sends the cumulative number of buffers mapped to the filter driver layer recorded in the host register, rather than the number of newly added buffers or the number of available buffers. The cumulative number of mapped buffers includes the number of buffers in the initial state of the unprocessed image and the number of buffers remapped to the filter driver layer after processing. The camera side needs to set a counter to accumulate the number of images sent in this round. The data recorded by the counter is continuously accumulated in units of images. Before each image is sent, the camera needs to compare the number of images sent recorded by the counter with the cumulative number of mapped buffers updated in the notification sent by the host to decide whether to send image data to the host. In this way, even if a notification fails to reach the camera, as long as a subsequent pass can successfully reach the camera and update the data, the synchronization between the camera and the host will not be affected. In this embodiment, since the counter in the camera is continuously accumulated, in order to prevent overflow, a 64-bit counter can be selected, which ensures that there is no risk of looping after the counter is full, and the counter will be reset synchronously after the acquisition is stopped. In addition, in order to further ensure the success rate of the notification, the host needs to send two notifications to the camera each time.

[0060] If an acknowledgment design is used between the host and the camera, that is, the camera must confirm with the host every time it receives a notification. Since the current industrial image acquisition system sends control commands based on UDP, this unreliable underlying transmission method may still experience packet loss or delayed arrival when the upstream bandwidth is heavily occupied (the upstream is usually full-bandwidth image transmission). Using the acknowledgment method instead increases the risk of notification failure, and may also trigger a retransmission storm in the case of high frame rate, acknowledgment loss, or timeout. Therefore, in this application, an unacknowledged method is preferably used.

[0061] After the processed buffer is put back into the acquisition queue and mapped to the filter driver layer, the current cumulative number of mapped buffers changes. The host application layer needs to send the current cumulative number of mapped buffers to the camera again. The camera then determines whether there are available buffers in the host application layer based on the current cumulative number of mapped buffers and the number of images already sent. The specific steps are as follows:

[0062] Step 4.1, the host application layer determines the frequency of sending the current cumulative number of mapped buffer notifications to the camera according to the total number of allocated buffers. Among them, the frequency of sending notifications decreases as the total number of buffers increases. The relationship between the frequency and the total number of buffers is:

[0063]

[0064] In the formula, f is the frequency of the host application layer sending the current cumulative number of mapped buffer notifications to the camera, and its unit is the number of times a buffer is mapped to the filter driver layer. N is the total number of allocated buffers. For example, when N < 50, f = 1, which means that whenever a buffer is mapped to the filter driver layer, the host application layer will send the current cumulative number of mapped buffers to the camera. When 50 < N < 100, f = 2, which means that whenever two buffers are mapped to the filter driver layer, the host application layer will send the current cumulative number of mapped buffers to the camera.

[0065] In this embodiment, although the use of an unanswered notification method between the host and the camera can avoid the retry storm caused by the loss of the response to a certain extent, when the industrial camera works in the small window mode, the image acquisition will reach a very high frame rate. If the number of available buffers is synchronized once for each frame of image sent to the host, that is, the host sends a notification of the cumulative number of mapped buffers to the camera, it will still cause the notification interaction frequency to be too high; and in the case of a higher frequency of notifications, the embedded firmware program on the camera side is also difficult to respond to frequent interrupts. At the same time, the high frequency notification will also affect other interrupts of the camera. For this reason, the present application performs special processing on the notification of the cumulative number of mapped buffers; specifically, on the camera side, after receiving the notification, the camera uses FPGA (a hardware device) to filter it to prevent it from reaching the camera firmware and prevent the notification from causing a firmware interrupt in the camera. Since the data transmission and processing in the camera are completed by FPGA, a higher notification efficiency can be maintained. On the host side, since the number of buffers will be automatically adjusted according to the size of the captured image payload when allocating buffers, the smaller the image, the more buffers will be allocated. Therefore, the frequency of sending notifications can be adjusted according to the number of allocated buffers. When the number of allocated buffers is large, the frequency of notifications will be reduced, and when the number of allocated buffers is small, the frequency of notifications will be increased. In this way, the frequency of notifications will not increase suddenly due to the high frame rate of small images. Since the camera side directly filters notifications through hardware, the camera firmware will not be interrupted frequently.

[0066] Step 4.2, the host application layer sends the current cumulative mapping buffer number notification to the camera at frequency f, and starts the timer synchronously after the sending is completed. If the cumulative mapping buffer number remains unchanged within the preset time, the host application layer sends the current cumulative mapping buffer number notification to the camera again. If the cumulative mapping buffer number increases within the preset time, the timer is reset after the next notification is sent;

[0067] When the timer reaches the preset value and the cumulative number of mapping buffers remains unchanged, the host application layer determines that multiple consecutive notifications have not correctly reached the camera, and the industrial image acquisition system enters a deadlock state, or determines that the user takes too long to process the image. At this time, the camera has not received any notifications, and the host application layer actively sends a notification of the cumulative number of mapping buffers to the camera, where the cumulative number of mapping buffers sent this time is the same as the previous time.

[0068] In this embodiment, during the notification interaction between the host application layer and the camera, if a certain notification cannot reach the camera correctly, synchronous error correction can be achieved only when the next notification can reach the camera. However, there are two special situations. One is that multiple consecutive notifications from the host application layer (referring to the notification of the current cumulative number of mapping buffers sent by the host application layer to the camera) have not reached the camera correctly. The camera has not received the updated cumulative number of mapping buffers, and will determine that the host application layer has no available buffers and will no longer send images to the host. Since there are no images to be processed in the host, the cumulative number of mapping buffers in the register will not be updated again. At this time, the industrial image acquisition system enters a deadlock state; the other is that the user processes the image for too long, resulting in all buffers in the acquisition queue being placed in the processing queue, and all buffers have not been returned. In order to avoid the industrial image acquisition system from entering a deadlock state, a timed sending mechanism needs to be added after the host application layer sends the notification. After the industrial image acquisition system enters a deadlock state, the host application layer actively sends a notification to the camera to inform the camera of the current cumulative number of mapping buffers.

[0069] Under normal circumstances, the processing speed of the visual system should be greater than the acquisition speed. After processing each image, the host returns the buffer to the host application layer, at which point the host application layer re-adds it to the acquisition queue and sends a notification to the camera. Each time the transmission is completed, a timer is started synchronously. If the host application layer does not receive the processed buffer within the specified time, it will actively send a notification of the cumulative number of mapped buffers to the camera. However, because there is no free buffer to return at this time, the cumulative number of mapped buffers sent this time is the same as the previous time.

[0070] Step 4.3, the camera extracts the notification of the cumulative number of mapped buffers through the FPGA module, obtains the number of images sent, compares the current cumulative number of mapped buffers with the number of images sent, and if the current cumulative number of mapped buffers is greater than the number of images sent, determines that the host application layer has available buffers; if the current cumulative number of mapped buffers is less than or equal to the number of images sent, determines that the host application layer has no available buffers;

[0071] The camera filters notifications about the cumulative number of mapped buffers through the FPGA module to prevent them from reaching the camera firmware, obtains information about the cumulative number of mapped buffers, reads the number of sent images currently recorded by the camera counter, and compares the cumulative number of mapped buffers with the number of sent images. If the current cumulative number of mapped buffers is greater than the number of sent images, it is determined that the host application layer has available buffers, which means that the camera can continue to send images to the host. If the current cumulative number of mapped buffers is less than or equal to the number of sent images, it is determined that the host application layer has no available buffers, which means that the camera needs to stop sending images to the host to avoid image data loss.

[0072] Step 5: The camera continues to send images to the host, and the host application layer continues to receive images through the acquisition queue and puts the buffer into the processing queue to notify the user to process. After the processing is completed, it returns to step 3.

[0073] The camera continues to send a new frame of image. The host application layer receives the current new frame of image through the acquisition queue and stores it in the corresponding buffer. Then, the buffer that has completed storing the image in the acquisition queue is placed in the processing queue. The user is notified through callback to process it. After the user completes the processing, the current buffer is placed back into the acquisition queue and mapped to the filter driver layer again.

[0074] Step 6: The camera stops sending images to the host and caches the currently acquired image data into the camera's buffer. When the current cumulative number of mapped buffers is greater than the number of images sent, the camera returns to step 5.

[0075] After pausing to send images, the camera caches the currently acquired image data into the camera's buffer and waits for the next notification from the host. If after the next notification, the cumulative number of mapped buffers is greater than the number of images sent, the camera determines that the host application layer has available buffers and can continue to send images to the host, and returns to step 5. If after the next notification, the cumulative number of mapped buffers is still less than or equal to the number of images sent, the camera determines that the host application layer has no available buffers and cannot send images to the host, and the camera continues to wait for the host's notification of the cumulative number of mapped buffers.

[0076] Repeat steps 3 to 6 until the camera stops acquiring images. After the camera stops acquiring images, close the camera's counter and the host's register.

[0077] The application layer flow control method based on image network transmission in the present application directly establishes a data buffer overflow warning feedback mechanism between the host application layer and the industrial camera, and can achieve the purpose of image transmission flow control through the unresponsive notification method between the host and the camera. On the host side, the host's application layer will maintain an acquisition queue, and the host application layer can feed back real-time information of the acquisition queue to the camera. When the camera determines that the host has no buffer for data reception, it can suspend data transmission and temporarily cache the data collected from the sensor in the camera's buffer, thereby completing the flow control from the data source to the application layer. Moreover, this control is based on image frames, which effectively solves the problem of discontinuity of the buffer overflow warning feedback mechanism in the current network interface industrial image acquisition chain, and can make full use of the buffer inside the camera to eliminate the impact of instantaneous data blocking, greatly improving the stability of the network interface industrial image acquisition system.

[0078] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0079] The units in the device of the present application can be combined, divided and deleted according to actual needs.

[0080] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. An application layer flow control method based on image network transmission, which is applied to the process of industrial cameras transmitting image data to a host through a network interface. The process uses a layer-by-layer back pressure mechanism to control data transmission flow. The host side includes a network card driver and a filter driver. The network card driver is used to transmit the received image data to the filter driver. It is characterized in that: The method includes: Step 1: Use the camera SDK to allocate a group of buffers at the host application layer, and create a corresponding acquisition queue for receiving image data from the industrial camera. Add the buffer to the acquisition queue as a data receiving unit, and map the buffer to the host's filter driver layer, where the mapping is to pass the buffer's descriptor to the filter driver layer, so that the filter driver layer can transfer the image to the acquisition queue; Step 2: The host application layer establishes direct communication with the camera via the network card, sends the cumulative mapping buffer number to the camera, and the camera starts to collect images and sends them to the host. The host application layer receives the images through the collection queue and puts the buffer into the processing queue to notify the user to process; Step 3: The host application layer puts the processed buffer back into the acquisition queue and maps it to the filter driver layer; Step 4: The host application layer sends the cumulative number of mapping buffers to the camera. The camera checks the current cumulative number of mapping buffers and the number of images sent. If the current cumulative number of mapping buffers is greater than the number of images sent, execute step 5. If the current cumulative number of mapping buffers is less than or equal to the number of images sent, execute step 6. Step 5: The camera continues to send images to the host, and the host application layer continues to receive images through the acquisition queue and puts the buffer into the processing queue to notify the user to process. After the processing is completed, it returns to step 3. Step 6: The camera stops sending images to the host and caches the currently acquired image data into the camera's buffer. When the current cumulative number of mapped buffers is greater than the number of images sent, the camera returns to step 5.

2. The application layer flow control method based on image network transmission according to claim 1, characterized in that: The host sets a register, and step 2 specifically includes: The host application layer obtains the cumulative number of mapped buffers mapped to the filter driver layer through the register and sends a notification to the camera. The camera starts to collect images and sends them to the host frame by frame. The images enter the host side network card through the network cable, and then pass through the network card driver layer and the filter driver layer in sequence, and finally enter the host application layer. The host application layer receives the image through the acquisition queue and stores the image in the corresponding buffer, and then puts the buffer in the acquisition queue that has completed image storage into the processing queue, and notifies the user to process it.

3. The application layer flow control method based on image network transmission according to claim 2, characterized in that: The buffer processed in step 3 is released, and the host application layer marks the released buffer as available, puts it back into the acquisition queue, and maps it to the filter driver layer again. At the same time, the register updates the accumulated number of mapped buffers.

4. The application layer flow control method based on image network transmission according to claim 3, characterized in that: The step 4 specifically includes: Step 4.1, the host application layer determines the frequency of sending notifications of the current cumulative number of mapped buffers to the camera according to the total number of allocated buffers, wherein the frequency of sending notifications decreases as the total number of buffers increases; Step 4.2, the host application layer sends a notification of the current cumulative number of mapping buffers to the camera at the frequency, and starts a timer synchronously after the sending is completed. If the cumulative number of mapping buffers remains unchanged within the preset time, the host application layer sends a notification of the current cumulative number of mapping buffers to the camera again. If the cumulative number of mapping buffers increases within the preset time, the timer is reset after the next notification; In step 4.3, the camera extracts the notification of the cumulative number of mapped buffers through the FPGA module, obtains the number of images sent, and compares the current cumulative number of mapped buffers with the number of images sent. If the current cumulative number of mapped buffers is greater than the number of images sent, it is determined that the host application layer has available buffers. If the current cumulative number of mapped buffers is less than or equal to the number of images sent, it is determined that the host application layer has no available buffers.

5. The application layer flow control method based on image network transmission according to claim 4, characterized in that: The relationship between the frequency and the total number of buffers is: Wherein, f is the frequency at which the host application layer sends notifications to the camera, the unit is the number of times the buffer is mapped to the filter driver layer, and N is the total number of allocated buffers.

6. The application layer flow control method based on image network transmission according to claim 4, characterized in that: In step 4.2, when the timer reaches the predetermined value and the cumulative number of mapping buffers remains unchanged, the host application layer determines that multiple consecutive notifications have not reached the camera, and the industrial image acquisition system enters a deadlock state, or determines that the user takes too long to process the image, and the host application layer actively sends a notification of the cumulative number of mapping buffers to the camera, wherein the cumulative number of mapping buffers sent is the same as the previous time.

7. The application layer flow control method based on image network transmission according to claim 2, characterized in that: In step 6, the camera caches the currently acquired image data into the camera's buffer and waits for the next notification from the host. If after the next notification, the cumulative number of mapped buffers is greater than the number of images sent, the camera determines that the host application layer has available buffers and returns to step 5. If after the next notification, the cumulative number of mapped buffers is less than or equal to the number of images sent, the camera determines that the host application layer has no available buffers and continues to wait for notification.

8. The application layer flow control method based on image network transmission according to claim 1, characterized in that: The step 1 specifically includes: Import the camera SDK in the host application layer, obtain the camera's device information through the interface API provided by the camera SDK, establish a communication link with the camera based on the device information, use the camera SDK to allocate a set of buffers for storing images in the memory of the host application layer, and then define an acquisition queue for receiving images. Pass all buffer descriptors to the acquisition queue so that the acquisition queue can recognize the buffer and store the image data in the buffer after receiving it. Finally, map the buffer in the acquisition queue to the host's driver layer so that the filter driver layer can smoothly transfer the image to the acquisition queue.

Images