Video data stream processing method, image processing method, equipment and storage medium

Abstract

The invention provides a video data stream processing method, an image processing method, equipment and a storage medium, which can be applied to the technical field of artificial intelligence, especially the technical fields of computer vision, streaming media and the like. The video data stream processing method comprises the following steps: determining N video data streams from N image acquisition devices; wherein each video data stream comprises at least one data packet, and N is an integer greater than or equal to 2; dividing a plurality of data packets included in the N video data streams into N data packet sets, wherein each data packet set corresponds to one video data stream; determining N first queues respectively corresponding to the N data packet sets, wherein each first queue comprises a first storage address of each data packet in the data packet set corresponding to the first queue in the first storage space; and sending the N first queues.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as computer vision and streaming media. Background Art

[0002] An image acquisition device can be used to acquire video stream data, and then image frames in the video stream data can be processed to obtain image processing results. For example, multiple video data streams acquired by multiple image acquisition devices are processed independently, and each video data stream occupies a thread, and a video stream data is decoded, and object detection is performed separately. However, this processing method has the problems of low efficiency and low precision. Summary of the invention

[0003] The present disclosure provides a video data stream processing method, an image processing method, a video data stream processing apparatus, an image processing apparatus, a video data stream processing device, an image processing device, an image processing system, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, there is provided a method for processing a video data stream, comprising: determining N video data streams from N image acquisition devices; wherein each video data stream comprises at least one data packet, and N is an integer greater than or equal to 2; dividing the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to one video data stream; determining N first queues corresponding to the N data packet sets, respectively, each first queue comprising a first storage address in a first storage space of each data packet in the data packet set corresponding to the first queue; and sending the N first queues.

[0005] For example, dividing multiple data packets included in N video data streams into N data packet sets includes: determining the video identifier of the video data stream in which each data packet is located; dividing the multiple data packets according to the video identifier to obtain N initial data packet sets; and determining N data packet sets according to the N initial data packet sets.

[0006] For example, according to N initial data packet sets, determining N data packet sets includes: sorting the data packets in the initial data packet set according to the sequence numbers of the data packets in the initial data packet set to obtain a data packet sequence; and determining the data packet set according to the data packet sequence.

[0007] For example, a data packet set includes at least one data packet subset, each data packet subset includes at least one data packet, and each data packet subset corresponds to an image frame; determining N first queues corresponding to the N data packet sets respectively includes: in response to detecting that the first queue includes a first storage address for a previous data packet, adding the first storage address for the current data packet to the first queue; wherein the current data packet and the previous data packet belong to the same data packet subset; and in response to detecting that the first queue includes a first storage address for a previous data packet subset, adding the first storage address for the current data packet subset to the first queue; wherein the current data packet subset and the previous data packet subset belong to the same data packet set.

[0008] For example, the data packet set includes at least one data packet subset, each data packet subset corresponds to an image frame; sending N first queues includes: in response to detecting that the target storage address in the first queue meets a predetermined condition, sending the target storage address; wherein the predetermined condition includes: the data packet indicated by the target storage address is used to determine an image frame.

[0009] For example, the predetermined condition also includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold.

[0010] For example, the method further includes: in response to the plurality of target data packets in the first storage space being read, deleting the plurality of target data packets from the first storage space.

[0011] For example, it also includes: determining a target adjustment parameter based on a scene identifier of a target scene; the target adjustment parameter includes at least one of the following: a target number of the image acquisition device and a target acquisition frequency of the image acquisition device; in response to detecting that the target adjustment parameter includes the target number of the image acquisition device, adjusting the actual number of image acquisition devices in use according to the target number; and in response to detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition device, adjusting the actual acquisition frequency of the image acquisition device according to the target acquisition frequency.

[0012] For example, a data packet includes a network layer abstraction unit.

[0013] According to another aspect of the present disclosure, there is provided an image processing method, comprising: determining N second queues based on N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in a second storage space; determining a plurality of data packets from the second storage space based on the second storage addresses in the N second queues; determining a plurality of image frames based on the plurality of data packets; and processing the plurality of image frames using a predetermined algorithm to obtain processing results.

[0014] For example, based on N first queues, determining N second queues includes: reading multiple target data packets from the first storage space based on the first storage address received in the first queue; storing the multiple target data packets in the second storage area; and adding the second storage addresses of the multiple target data packets in the second storage area to the second queue.

[0015] According to another aspect of the present disclosure, a video data stream processing device is provided, comprising: a video data stream determination module, a division module, a first queue determination module and a sending module. The video data stream determination module is used to determine N video data streams from N image acquisition devices; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2. The division module is used to divide the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to a video data stream. The first queue determination module is used to determine N first queues corresponding to the N data packet sets, respectively, and each first queue includes the first storage address of each data packet in the data packet set corresponding to the first queue in the first storage space. The sending module is used to send the N first queues.

[0016] According to another aspect of the present disclosure, an image processing device is provided, comprising: a second queue determination module, a data packet determination module, an image frame determination module and a processing module. The second queue determination module is used to determine N second queues based on N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in the second storage space. The data packet determination module is used to determine multiple data packets from the second storage space based on the second storage addresses in the N second queues. The image frame determination module is used to determine multiple image frames based on multiple data packets. The processing module is used to process multiple image frames using a predetermined algorithm to obtain processing results.

[0017] According to another aspect of the present disclosure, a video data stream processing device is provided, including a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the above-mentioned video data stream processing method.

[0018] According to another aspect of the present disclosure, an image processing device is provided, including a memory and a processor, wherein instructions executable by the processor are stored in the memory, and when the instructions are executed by the processor, the processor executes the above image processing method.

[0019] According to another aspect of the present disclosure, there is provided an image processing system, comprising: N image acquisition devices for acquiring N video data streams, the above-mentioned video data stream processing device and the above-mentioned image processing device.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a schematic flow chart of a video data stream processing method according to an embodiment of the present disclosure;

[0025] Figure 2 is a schematic flow chart of an image processing method according to an embodiment of the present disclosure;

[0026] Figure 3 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

[0027] Figure 4 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

[0028] Figure 5 is a schematic diagram of the structure of multiple data packets according to an embodiment of the present disclosure;

[0029] Figure 6 is a schematic diagram of a data structure of a data packet according to an embodiment of the present disclosure;

[0030] Figure 7 is a schematic diagram of data packet extraction according to an embodiment of the present disclosure;

[0031] Figure 8 is a schematic diagram of a processing process of an image acquisition device according to an embodiment of the present disclosure;

[0032] Fig. 9 is a schematic structural diagram of an image processing device according to another embodiment of the present disclosure;

[0033] Fig.10 is a schematic structural block diagram of a video data stream processing device according to an embodiment of the present disclosure;

[0034] Fig.11is a schematic structural block diagram of an image processing device according to an embodiment of the present disclosure; and

[0035] Fig.12 It is a structural block diagram of an electronic device used to implement the video data stream processing method and / or image processing method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0037] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0038] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0039] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0040] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0041] The present disclosure provides a video data stream processing method and an image processing method. The video data stream processing method can be executed by a video data stream processing device, which is a source. The image processing method can be executed by an image processing device, which is a sink.

[0042] During the processing, the video data stream processing device can obtain the video data stream collected by the image acquisition device, and then determine N first queues based on the video data stream, and send the N first queues to the image processing device. Next, the image processing device can determine the image frame based on the N first queues, and then process the image frame to obtain the processing result.

[0043] The technical solution provided by the embodiments of the present disclosure achieves efficient access to multiple streaming media by performing processes such as splicing and extraction on data packets, determines multiple image frames based on multiple video stream data collected by multiple image acquisition devices, and then performs batch processing on the multiple image frames, thereby solving the problem of inefficiency in multi-threaded processing of multiple streaming media, as well as the problem of decreased overall accuracy and increased false alarm rate of computer vision applications caused by the inefficiency, and can also reduce equipment load, reduce costs, and improve equipment utilization.

[0044] The technical solution provided in the embodiments of the present disclosure is applicable to streaming media computer vision applications. In the actual deployment process, the solution provided in the present disclosure is applicable to, for example, a server (e.g., an x86 server), a box (e.g., an ARM box), Android, etc., and can be deployed on terminal devices such as televisions and mobile phones.

[0045] The video data stream processing method and the image processing method are described below.

[0046] Figure 1 It is a schematic flow chart of a video data stream processing method according to an embodiment of the present disclosure.

[0047] like Figure 1 As shown, the video data stream processing method 100 may be executed by a video data stream processing device, and the method 100 includes operations S110 to S140.

[0048] In operation S110, N video data streams from N image acquisition devices are determined; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2.

[0049] For example, the image acquisition device may include a camera, such as a camera installed in a park, a shopping mall, or a camera installed on a moving object such as an autonomous vehicle. N may be 4, 16, or the like.

[0050] For example, the image acquisition device may use a video data stream, and the N image acquisition devices correspond one to one with the N video data streams. For example, the first image acquisition device Cam1 acquires the first video data stream Video1, and the second image acquisition device Cam2 acquires the second video data stream Video2.

[0051] It should be noted that, in the actual processing process, the video data stream collected by the image acquisition device may be an H.264 code stream, which may be RTSP encapsulated and then the data packets in the video data stream may be obtained by decompressing the RTSP and H.264 code stream.

[0052] In operation S120, a plurality of data packets included in the N video data streams are divided into N data packet sets, each data packet set corresponding to one video data stream.

[0053] It should be noted that the data packets in the N data streams enter the first storage space in chronological order, so the multiple data packets in the first storage space are out of order. The multiple data packets in the first storage space can be divided to obtain N data packet sets.

[0054] For example, multiple data packets in a first video data stream from a first image acquisition device Cam1 are divided into a data packet set A1, and multiple data packets in a video data stream from a second image acquisition device Cam2 are divided into a second data packet set A2.

[0055] It should be noted that a video data stream collected by an image acquisition device can determine a data packet set, a data packet set can include at least one data packet subset, a data packet subset can include at least one data packet, and the data packets in a data packet subset can be used to determine a complete image frame. For example, the data packet set determined by the video data stream Video1 can include data packets package1, package2, package3, package4, package5, package6 and package7. The data packet set can be a first data packet subset and a second data packet subset. The first data packet subset can include data packets package1, package2, package3. The image processing device can use the first data packet set to determine an image frame, and the second data packet subset can include package4, package5, package6 and package7. The image processing device can use the second data packet set to determine another image frame. The above-mentioned data packets are, for example, data packets of the RTP protocol, for example, multiplexed packets compressed by H264.

[0056] It should be noted that, in the process of transmitting a video data stream, if a piece of data that can determine a complete image frame is recorded in a data packet, the data volume of the data packet is relatively large. In the actual transmission process, if the data packet transmission fails due to network anomalies and other reasons, all the data that can determine a complete image frame needs to be retransmitted. However, if the data that can determine a complete image frame is divided into multiple data packets (the multiple data packets form a data packet subset), the data packet can be retransmitted after a data packet transmission fails, without transmitting the complete data packet subset, thereby improving the transmission efficiency.

[0057] In operation S130, N first queues corresponding to the N data packet sets are determined, each of the first queues including a first storage address in the first storage space of each data packet in the data packet set corresponding to the first queue.

[0058] For example, the first storage space may be a memory of a video data stream processing device, and the storage address of the data packet in the first storage space is referred to as a first storage address.

[0059] For example, N data packet sets correspond one-to-one to N first queues, for example, data packet set A1 corresponds to the first queue Q1, and data packet set A2 corresponds to the first queue Q2. Therefore, the first storage address of the data packet in data packet set A1 can be added to the first queue Q1, and the first storage address of the data packet in data packet set A2 can be added to the first queue Q2.

[0060] In operation S140 , N first queues are transmitted.

[0061] For example, N first queues are sent to an image processing device, and the transmission mode of the first queues may be a wired transmission mode such as PCIE, or a wireless transmission mode such as WiFi.

[0062] The technical solution provided by the embodiment of the present disclosure determines N first queues based on N video data streams, and each first queue includes the first storage addresses of multiple data packets from an image acquisition device. During the processing of the image processing device, multiple image frames can be determined based on the first queue, so that the multiple image frames can be processed in batches, thereby making better use of the computing resources of the image processing device and improving computing efficiency and accuracy.

[0063] In addition, the information in the first queue is the first storage address of the data packet, not the data packet itself. Therefore, during data transmission, the data packet only needs to be copied once, and there is no need to copy the data table to the first queue, thereby reducing the amount of data transmission, improving data transmission efficiency, and reducing the requirements for device hardware.

[0064] According to another embodiment of the present disclosure, the above-mentioned process of dividing the multiple data packets included in N video data streams into N data packet sets may include: determining the video identifier of the video data stream in which each data packet is located, and then dividing the multiple data packets according to the video identifier to obtain N initial data packet sets, and then determining N data packet sets based on the N initial data packet sets.

[0065] For example, a data packet may include a video identifier, and multiple data packets are divided based on the video identifier to obtain multiple initial data packet sets, each of which corresponds to the same video identifier. For example, the video identifiers of multiple data packets in the first initial data packet set are all media1, and the video identifiers of multiple data packets in the second initial data packet set are all media2. This embodiment divides the data packet sets according to the video identifier, so that a first queue stores the first storage address for a data packet from an image acquisition device, thereby ensuring that N initial data packet sets correspond one-to-one to N video data streams.

[0066] For example, N initial data packet sets may be determined as N data packet sets, and the data packets in the data packet sets may be randomly ordered.

[0067] For another example, the data packets in the initial data packet set may be sorted according to the serial numbers of the data packets in the initial data packet set to obtain a data packet sequence, and then the data packet set may be determined according to the data packet sequence.

[0068] For example, when transmitting data packets based on the RTP protocol or the TCP protocol, each data packet has a serial number, which is also called a sequence number. The serial number can be a long integer data, and multiple serial numbers can be continuous. The data packets in the data packet set are arranged in sequence according to the serial number.

[0069] In this embodiment, the data packets in the data packet set are arranged in sequence according to the serial number. Compared with the random arrangement, the technical solution provided by this embodiment can ensure that the first storage address in the first queue is arranged in sequence according to the serial number of the data packet after the first queue is determined based on the data packet set, thereby ensuring that the image frame can be determined based on the ordered data packets later. In addition, the image processing device needs to decode the data packets in the order of the data packets during the processing process to avoid decoding errors. Since the data packets in the data packet set of this embodiment are arranged in the order of the serial number, the image processing device can decode the data packets in order, without the need to re-determine the decoding order based on the random data packets, thereby improving the processing efficiency of the image processing device and saving performance.

[0070] According to another embodiment of the present disclosure, in the process of determining N first queues corresponding to N data packet sets, for different data packet subsets in the same data packet set, after detecting that the first queue includes the first storage address for the previous data packet subset, the first storage address for the current data packet subset can be added to the first queue. In this way, the first storage addresses of multiple data packets in the same data packet subset can be added to the first queue in sequence, thereby ensuring the orderliness of the data packets.

[0071] According to another embodiment of the present disclosure, in the process of determining N first queues corresponding to N data packet sets, for different data packets in the same data packet subset, the same data packet subset includes a current data packet and at least one previous data packet, after detecting that the first queue includes a first storage address for the previous data packet, the first storage address for the current data packet can be added to the first queue. In this way, the first storage addresses of multiple data packet subsets in the same data packet set can be added to the first queue in sequence, thereby ensuring the orderliness of the data packet subsets.

[0072] According to another embodiment of the present disclosure, in the process of sending N first queues, the target storage address may be sent after it is detected that the target storage address in the first queue meets a predetermined condition.

[0073] For example, the predetermined conditions include: the data packet indicated by the target storage address is used to determine an image frame, the number of target storage addresses can be multiple, and the multiple data packets stored in the multiple target storage addresses can determine a complete image frame, thereby avoiding the image processing device being unable to obtain a complete image frame based on the data packet during subsequent processing, thereby ensuring the accuracy of the image frame processing results.

[0074] For another example, the predetermined condition includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold. The threshold may be an integer greater than or equal to 2, for example, the threshold is any value between 8 and 16. The number of image frames processed in batches by the image processing device may be controlled by the threshold, thereby avoiding wasting computing resources of the image processing device due to a small number of image frames, and ensuring data processing efficiency.

[0075] According to another embodiment of the present disclosure, the video data stream processing method may further include the following operations: determining a target adjustment parameter according to a scene identifier of a target scene, and then adjusting usage of N image acquisition devices according to the target adjustment parameter.

[0076] For example, the scene may be related to the data processing capability of the device. The machine may include at least one of a video data stream processing device and an image processing device. The data processing capability of the device may be determined based on hardware information such as the number of processor cores. The correspondence between device information and scenes may be preconfigured, so that after determining the device information of the currently used device, the scene identifier may be determined based on the correspondence.

[0077] For example, the correspondence between scenes and adjustment parameters may be configured in advance according to actual needs, and then after obtaining the scene identifier of the target scene, the adjustment parameters corresponding to the scene identifier are searched based on the scene identifier and the correspondence as the target adjustment parameters.

[0078] The target adjustment parameter may include the target number of image acquisition devices. For example, some devices have weak data processing capabilities and can only process video data streams acquired by 5 image acquisition devices, while other devices have strong data processing capabilities and can process video data streams acquired by 10 image acquisition devices. Therefore, after detecting that the target adjustment parameter includes the target number of image acquisition devices, the actual number of image acquisition devices in use may be adjusted according to the target number, for example, the adjusted actual number of image acquisition devices in use may be made consistent with the target number, so that the number of video data streams matches the processing capability of the device.

[0079] The target adjustment parameter may include the target acquisition frequency of the image acquisition device. For example, some devices have weak data processing capabilities and can process video data streams with fps (frames per second) of 12, while other devices have strong data processing capabilities and can process video data streams with fps of 25. Therefore, after detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition device, the actual acquisition frequency of the image acquisition device can be adjusted according to the target acquisition frequency, for example, the actual acquisition frequency of the image acquisition device after adjustment is made consistent with the target frequency, so that the acquisition frequency of the video data stream matches the processing capability of the device.

[0080] Figure 2 is a schematic flowchart of an image processing method according to an embodiment of the present disclosure.

[0081] like Figure 2 As shown, the image processing method 200 may be executed by an image processing device, and the method 200 may include operations S210 to S240.

[0082] In operation S210, N second queues are determined according to the N first queues, where N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in the second storage space.

[0083] For example, the video data stream processing device may adopt the above-mentioned video data stream processing method to obtain N first queues, and then send the N first queues to the image processing device.

[0084] In one example, the first storage address in the first queue indicates the address of the data packet in the first storage space, and the second storage address in the second queue indicates the address of the data packet in the second storage space. According to the received first storage address in the first queue, the image processing device can read multiple target data packets from the first storage space, then store the multiple target data packets in the second storage area, and then add the second storage addresses of the multiple target data packets in the second storage area to the second queue.

[0085] In another example, the image processing device and the video stream processing device share the same storage space, and in this case, the N first queues can be determined as N second queues.

[0086] In operation S220, a plurality of data packets are determined from the second storage space according to the second storage addresses in the N second queues.

[0087] For example, for each second queue, a data packet at the address may be read from the second storage space based on the second storage address in the second queue.

[0088] In operation S230, a plurality of image frames are determined according to the plurality of data packets.

[0089] For example, the read data packets are decoded and processed, and then a data packet in a data packet subset corresponding to an image frame is used to determine an image frame. By processing multiple data packet subsets in this way, multiple image frames can be obtained, and the multiple image frames are a batch of data.

[0090] In operation S240, a plurality of image frames are processed using a predetermined algorithm to obtain a processing result.

[0091] For example, a deep learning model may be used as the predetermined algorithm. The deep learning model includes YOLO, for example. Multiple images may be input into the YOLO model, and the model outputs a detection box, which may be used as a processing result.

[0092] The technical solution provided in this embodiment determines N first queues based on N video data streams, and each first queue includes the first storage addresses of multiple data packets from an image acquisition device. During the processing of the image processing device, multiple image frames can be determined based on the first queue, so that the multiple image frames can be processed in batches, thereby making better use of the computing resources of the image processing device and improving computing efficiency and accuracy.

[0093] In some embodiments, after the image processing device reads a plurality of target data packets from the first storage space, the video data stream processing device deletes the read plurality of target data packets from the first storage space so that the first storage space receives a new data packet.

[0094] Figure 3 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure.

[0095] like Figure 3 As shown, in this embodiment, the platform includes image acquisition devices 311 , 312 , 313 , a video data stream processing device 320 and an image processing device 330 .

[0096] The image acquisition devices 311 , 312 , 313 are used to acquire video data streams. The number of the image acquisition devices 311 , 312 , 313 may be N, where N is an integer greater than or equal to 2, and 3 are shown in the figure.

[0097] The video data stream processing device 320 can obtain N video data streams from N image acquisition devices 311, 312, and 313, each video data stream including at least one data packet, and the video data stream processing device 320 can divide multiple data packets in the N video data streams into N data sets and determine N first queues. Then, the N first queues can be sent to the image processing device 330 through the connector 340 or other methods. The connector 340 can be a PCIe (Peripheral Component Interconnect Expess) connector.

[0098] The image processing device 330 may determine N second queues based on the N first queues, and then the subsequent services may consume the second storage addresses in the second queues as consumers, thereby implementing the process of determining and processing image frames.

[0099] Figure 4 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure.

[0100] like Figure 4 As shown, in this embodiment, the platform includes an image acquisition device 410, a video data stream processing device 420 and an image processing device 430, and data is transmitted between the video data stream processing device 420 and the image processing device 430 via a connector 440. This embodiment mainly describes the data processing process of the video data stream processing device 420 and the data processing process of the image processing device 430 from a software perspective.

[0101] The number of the image acquisition devices 410 is N, which may include, for example, image acquisition devices 411 , 412 , and 413 . The N image acquisition devices 410 may transmit N video data streams to the video data stream processing device 420 .

[0102] The video data stream processing device 420 includes multiple functional modules, such as an identification generation module 421, a data packet combination module 422, a data packet splicing module 423, a data packet extraction module 424 and a first queue module 425. The functional modules in the video data stream processing device 420 are described below.

[0103] The identification generation module 421 obtains a video data stream from the image acquisition device 410. The video data stream includes multiple data packets. The identification generation module 421 generates a video identification for each data packet. The video identification indicates the identification of the video data stream to which the data packet belongs, and can also represent the image acquisition device 410 that sends the data packet. The video identification can be added to the extension field in the data packet. Next, the identification generation module 421 sends the video identification and the data packet to the data packet combination module 422. It should be noted that the identification generation module 421 can obtain the video data stream from the image acquisition device 410 in an asynchronous manner. The main reason for the asynchrony is network congestion. For example, the video data stream includes multiple data packets. The second data packet is delayed due to network congestion, resulting in the third data packet being sent first and the second data packet being sent later. At this time, the video data stream processing device 420 asynchronously receives the second data packet. The image processing device 430 uses an asynchronous method to process the video data stream, without waiting for the data packet that is temporarily unsuccessfully sent due to network congestion, thereby improving the transmission efficiency of the data packet.

[0104] The data packet combining module 422 stores the data packet in a first storage space, for example, in a memory of the video data stream processing device 420 .

[0105] The data packet splicing module 423 splices the multiple data packets in the first storage space into a time sequence stream with a video identifier. The data packets can be arranged according to the time sequence of the data packets received in the first storage space.

[0106] In the actual splicing process, the data packet set can form an N-dimensional array (ndarray) structure. Multiple data packets corresponding to the same video identifier can be first divided into the same initial data packet set, and then the multiple data packets in the initial data packet set are sorted according to the serial number. The sorted initial data packet set is the data packet set mentioned above. Next, a structural diagram of an N-dimensional array can be formed based on the order of the data packet set and the order of each data packet in the same data packet set. This structure is convenient for storing and using data packets. Since the data of the structure comes from N image acquisition devices 410, and a subset of data packets after splicing includes multiple (for example, 3) data packets, the N-dimensional array of these data packets can be conveniently operated and retrieved in rows and columns. In other embodiments, the data packet set can also form a linked list structure. It can be seen that if the data packet is a NALU (Network Abstract Layer Unit), the number of NALUs contained in the spliced ​​data structure will increase. The structure of the NALU data packet is described below and will not be repeated here.

[0107] It should be noted that if there is only one video data stream, the data packets in the first storage space only correspond to the one video data stream. When there are at least two video data streams, the data packets in the at least two video data streams enter the first storage space in chronological order, so the multiple data packets in the first storage space are out of order. For example, the order of the multiple video data packets in the first storage space can be: the first data packet in the first video data packet, the first data packet in the second video data packet, the second data packet in the first video data packet, the first data packet in the third video data packet, the third data packet in the first video data packet, etc.

[0108] In this embodiment, the data packet splicing module 423 is used to group and sort multiple data packets in the first storage space, and N data packet sets are obtained after sorting. The data packets in each data packet set are arranged in order, and the first queue can reflect the grouping and sorting of the data packets. In the subsequent processing process, the image processing device can process the data packets in order based on the first queue to obtain multiple image frames, and then batch process the multiple image frames, so as to make better use of the computing resources of the image processing device and improve the computing efficiency and accuracy.

[0109] The data packet extraction module 424 can extract the first storage address from the structure formed by the data packet. For example, the data packet includes multiple fields, wherein the data field indicates the specific data content in the data packet, and the first storage address can include the starting position and the ending position of the data field, or can include the starting position and the offset.

[0110] The first queue module 425 maintains N first queues. For a corresponding first queue and data packet set, the first storage address of a data packet in the data packet set can be added to the first queue.

[0111] The image processing device 430 includes a plurality of functional modules, for example, a second queue module 431, a depacketizing module 432, and a processing module 433. The functional modules in the image processing device 430 are described below.

[0112] The second queue module 431 maintains N second queues. The data structure of the first queue can be the same as the data structure of the second queue. The difference between the first queue and the second queue is that the first storage address in the first queue indicates the address of the data packet in the first storage space, and the first storage space is, for example, the CPU memory. The second storage address in the second queue indicates the address of the data packet in the second storage space, and the second storage space is, for example, the GPU video memory. Both the first queue and the second queue can be CV (computer vision) frame queues shared between processes.

[0113] The unpacking module 432 can obtain the second storage address from the second queue, and then read the data packet in the second storage space based on the second storage address, and then perform a predetermined decoding process, so as to process a single data packet subset into an image frame, so as to obtain multiple image frames (Frame), and the multiple image frames can be used as a batch (Batch) of data, and a batch of data can be used for AI business processing, alarm generation and other functions. It should be noted that the process of processing a single data packet subset into an image frame may include decoding, and this embodiment does not limit the decoding method.

[0114] In some embodiments, multiple image frames in a batch of data may be associated. For example, the batch of input data required for model processing for face recognition services may come from multiple image acquisition devices. When performing cross-border head tracking, the same object may appear in the acquisition field of view of some specific image acquisition devices. In this case, it is not necessary to use the data at all times as the input data of the model. Only the data packets for the object collected by these specific image acquisition devices can be used as a batch of data. In other embodiments, multiple image frames in a batch of data may also be unrelated.

[0115] The processing module 433 is used to process multiple image frames to obtain a processing result. The processing module 433 may include, for example, a CV pre-processing (CV RESIZE) module and a deep learning model. The deep learning model may be, for example, a YOLO model, which may include, for example, a convolutional layer. In the process of processing multiple image frames in a batch of data, the processing module 433 does not limit the order of the multiple image frames. The multiple image frames may be processed sequentially in a chronological order or in a random order.

[0116] It should be noted that the deep learning model can be determined according to actual needs, and this embodiment does not limit the deep learning model. The above-mentioned YOLO model can be applied to scenes of identifying objects in places such as parks and parks, and the objects may include users, vehicles, etc. The image processing device 430 can determine the source of the data packet based on the video identifier of the video to which the data packet belongs, and thus perform data analysis in combination with the actual position of the image acquisition device 410. For example, based on the identified user, the recognition accuracy of multiple people at various angles can be confirmed. Based on the identified vehicle, vehicle tracking can be performed. For example, the data packets of multiple image acquisition devices 410 can be spliced ​​into a group of image frames, which can improve the speed performance, and can also analyze the vehicle from multiple angles, thereby improving the accuracy of vehicle recognition.

[0117] Compared with the method of setting only one image acquisition device 410, this embodiment adopts N image acquisition devices 410, and the video data stream processing device 420 can output N first queues. Next, the image processing device 430 can process a batch of image frames. It should be noted that the processing module 433 can use a parallel processing method during the processing process. The time required for the processing model to process one image frame can be 201ns, and the time required to process eight image frames is only slightly more than 20ms. Therefore, for a predetermined number of image frames, more image frames can be processed in one processing process to use more computing resources of the image processing device 430, which can improve the processing efficiency of processing the predetermined number of image frames. It can be seen that this embodiment provides more image frames to the processing module 433 in one cycle of executing the code of the processing module 433 by increasing the batch size (BatchSize), batches the image frames, and reduces the computing amount and storage amount of the packet unpacking in each link of the data transmission, thereby improving the processing efficiency.

[0118] Figure 5 It is a schematic diagram of the structure of multiple data packets according to an embodiment of the present disclosure.

[0119] For example, the data packet splicing module mentioned above splices multiple data packets in the first storage space into a timing stream with a video identifier. It can be understood that each data packet subset in the data packet set in the first storage space is arranged in time sequence, and each data packet of the same data packet subset is arranged in serial number. The above timing stream can be represented by: a data packet set after multiple data packets are arranged in time sequence.

[0120] Figure 5 Two data packet sets are shown, each small square represents a data packet, one data packet set includes multiple data packet subsets, for example, one data packet subset includes data packets 1-1, 1-2 and 1-3, and another data packet subset includes data packets 1-4, 1-5 and 1-6. Another data packet set may also include multiple data packet subsets, each data packet subset includes at least one data packet, for example, a data packet subset includes data packet 2-1.

[0121] Figure 6 It is a schematic diagram of the data structure of a data packet according to an embodiment of the present disclosure.

[0122] like Figure 6 As shown, in some embodiments, the data packet may be a NALU. The data structure of the NALU is as follows: Figure 6 As shown, for example, the first bit is 0, the fourth to eighth bits represent the type, and the second 8 bits are the specific NALU data. Figure 6 A shaded area in the figure represents specific NALU data 601. After the specific NALU data 601 is an extension field 602, and information such as a video identifier and a serial number can be added to the extension field 602. The information such as the video identifier and the serial number can also be stored in other storage areas outside the data packet, and a mapping relationship between the above-mentioned information and the data packet is established according to the actual storage situation, and based on the mapping relationship, the video identifier, the serial number and the first storage address corresponding to each data packet can be queried.

[0123] Figure 7 is a schematic diagram of data packet extraction according to an embodiment of the present disclosure.

[0124] For example, the first storage address can be extracted from the structure formed by the data packet 7221. For example, a data packet has multiple fields, where the Data field represents the specific data content in the data packet, and the first storage address can include the starting position and the ending position of the Data field, or the starting position and the offset. Then the first storage address is added to the first queue 7251. For example, the first storage address of a data packet in another data packet set can be added to another first queue 7252. In addition, the data packet can also have an extension field, which can be used to store a video identifier and a serial number,

[0125] Figure 8 It is a schematic diagram of the processing process of the image acquisition device according to an embodiment of the present disclosure.

[0126] In the actual processing process, for example, the first process 901 of the video data stream processing device can add data packets 1 to 20 to the first queue 805. Among these data packets, data packets 1 to 9 are the first data packet subset, data packets 10 to 19 are the second data packet subset, and data packets 20 to 30 are the third data packet subset. The GPU of the image processing device has more cores and can run more processes in parallel. For example, the number of cores can be the same as the number of parallel threads, and one thread can include multiple processes. Figure 8 As shown, the first subset of packets can be pulled from the second queue 806 by the second process 802, the second subset of packets can be pulled from the second queue 806 by the third process 803, and the third subset of packets can be pulled from the second queue 806 by the fourth process 804. The second process 802, the third process 803, and the fourth process 804 can be processed in parallel.

[0127] Fig. 9 is a schematic structural diagram of an image processing device according to another embodiment of the present disclosure.

[0128] like Fig. 9 As shown, in this embodiment, the platform includes image acquisition devices 911, 912, 913, a video data stream processing device 920, an image processing device 930, a PCIe bus 940 and a network card 950. This embodiment mainly describes the platform from the perspective of hardware.

[0129] N image acquisition devices are used to acquire video data streams, such as rtsp streams, which include rtp packets, and the rtp packets include H.264 NALUs. The video data stream is first transmitted by the image acquisition device to the network card 950, and then transmitted to the processor 921 in the video data stream processing device 920 via the PCIe bus 940.

[0130] The video data stream processing device 920 includes at least one processor 921, each processor may include at least one computing unit 9211, and the processor 921 includes 4 computing units 9211, which means that the number of cores of the processor 921 is 4. The computing unit 9211 can process the video data stream, and the video data stream includes multiple data packets, and the data packets can be NALUs. The video data stream processing device 920 processes multiple data packets and can obtain N first queues. Next, multiple data packets in the N first queues and / or the N video data streams are transmitted to the image processing device 930 via the PCIe bus 940, and the image processing device 930 processes multiple data packets to determine image frames and processing results for the image frames.

[0131] It should be noted that Figure 4 The data processing process of the video data stream processing device is mainly explained from the software perspective. Fig. 9 The video data stream processing equipment is mainly explained from the hardware perspective. Fig. 9 The process of processing data packets by the computing unit 9211 can be referred to Figure 4 The data processing process of the identification generation module 421, the data packet combination module 422, the data packet splicing module 423, the data packet extraction module 424 and the first queue module 425 in the embodiment will not be repeated here.

[0132] It should be noted that Fig. 9 Four processors 921 and four computing units 9211 are shown, wherein the processor 921 including the computing unit 9211 at the bottom is used to illustrate one of the processors 921 at the top, and the computing unit 9221 at the bottom is used to illustrate the first computing unit 9221 at the top.

[0133] Fig.10 It is a schematic structural block diagram of a video data stream processing device according to an embodiment of the present disclosure.

[0134] like Fig.10 As shown, the video data stream processing apparatus 1000 may include a video data stream determining module 1010 , a dividing module 1020 , a first queue determining module 1030 and a sending module 1040 .

[0135] The video data stream determination module 1010 is used to determine N video data streams from N image acquisition devices, wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2. For example, the video data stream determination module 1010 can be used to implement a portion of the functions of the identification generation module 421 described above.

[0136] The division module 1020 is used to divide the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to a video data stream. For example, the implementation of the division module 1020 can refer to the data packet combination module 422, the data packet splicing module 423 and the data packet extraction module 424 above.

[0137] The first queue determination module 1030 is used to determine N first queues corresponding to the N data packet sets, each of which includes a first storage address of each data packet in the data packet set corresponding to the first queue in the first storage space. The implementation of the first queue determination module 1030 can refer to the first queue module 425 above.

[0138] The sending module 1040 is used to send N first queues.

[0139] According to another embodiment of the present disclosure, the division module includes: an identification determination submodule, a division submodule and a data packet set determination submodule. The identification determination submodule is used to determine the video identification of the video data stream where each data packet is located. The division submodule is used to divide multiple data packets according to the video identification to obtain N initial data packet sets. The data packet set determination submodule is used to determine N data packet sets according to the N initial data packet sets.

[0140] According to another embodiment of the present disclosure, the data packet set determination submodule includes: a sequence determination unit and a data packet set determination unit. The sequence determination unit is used to sort the data packets in the initial data packet set according to the sequence numbers of the data packets in the initial data packet set to obtain a data packet sequence; the data packet set determination unit is used to determine the data packet set according to the data packet sequence. For example, the implementation of the sequence determination unit and the data packet set determination unit can refer to Figure 4 The data packet splicing module 423 in.

[0141] According to another embodiment of the present disclosure, a data packet set includes at least one data packet subset, each data packet subset includes at least one data packet, and each data packet subset corresponds to an image frame; the first queue determination module includes: a first adding submodule and a second adding submodule. The first adding submodule is used to add the first storage address for the current data packet to the first queue in response to detecting that the first queue includes a first storage address for the previous data packet; wherein the current data packet and the previous data packet belong to the same data packet subset. In this way, the first storage addresses of multiple data packets in the same data packet subset can be added to the first queue in sequence, thereby ensuring the orderliness of the data packet subset, that is, ensuring the orderliness of the image frame. The second adding submodule is used to add the first storage address for the current data packet subset to the first queue in response to detecting that the first queue includes a first storage address for the previous data packet subset; wherein the current data packet subset and the previous data packet subset belong to the same data packet set. In this way, the first storage addresses of multiple data packet subsets in the same data packet set can be added to the first queue in sequence, thereby ensuring the orderliness of the data packet subset, so that the subsequent image processing device can process the data packet on demand.

[0142] According to another embodiment of the present disclosure, the data packet set includes at least one data packet subset, each data packet subset corresponds to an image frame; the sending module includes: a sending submodule, used to send the target storage address in response to detecting that the target storage address in the first queue meets a predetermined condition; wherein the number of target storage addresses can be at least one, and the predetermined condition includes: at least one data packet indicated by the at least one target storage address is used to determine a complete image frame.

[0143] According to another embodiment of the present disclosure, the predetermined condition further includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold value. For example, the serial number of the data packet can reflect whether the data packet is the last data packet required for decoding to obtain the image frame, so the number of complete image frames corresponding to multiple data packets in the N data packet sets can be determined by the serial number.

[0144] For example, the threshold value may be preconfigured according to the size of the image frame. For example, the preset data volume for a single transmission is 20M, and the data volume for decoding an image frame is approximately 2M, then the threshold value may be set to 10.

[0145] For example, the threshold may be 1, which indicates real-time transmission, that is, when a data packet for decoding to obtain a complete image frame is received, the first queue having the first storage address storing the data packets may be sent.

[0146] For another example, the threshold may be greater than 1, which indicates batch transmission, that is, the first queue is transmitted only after the number of complete image frames corresponding to the data packets reaches the threshold.

[0147] According to another embodiment of the present disclosure, the system further includes: a deletion module, configured to delete multiple target data packets from the first storage space in response to multiple target data packets being read from the first storage space. It should be noted that in this embodiment, the data packets read from the first storage space are referred to as target data packets.

[0148] According to another embodiment of the present disclosure, it also includes: a parameter determination module, a quantity adjustment module and a frequency module. The parameter determination module is used to determine the target adjustment parameter according to the scene identification of the target scene; the target adjustment parameter includes at least one of the following: the target number of image acquisition devices and the target acquisition frequency of the image acquisition device; the quantity adjustment module is used to adjust the actual use number of image acquisition devices according to the target number in response to detecting that the target adjustment parameter includes the target number of image acquisition devices; the frequency module is used to adjust the actual acquisition frequency of the image acquisition device according to the target acquisition frequency in response to detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition device.

[0149] According to another embodiment of the present disclosure, the data packet includes a network layer abstraction unit.

[0150] Fig.11 is a schematic structural block diagram of an image processing device according to an embodiment of the present disclosure; and

[0151] like Fig.11 As shown, the image processing device 1100 may include a second queue determination module 1110 , a data packet determination module 1120 , an image frame determination module 1130 and a processing module 1140 .

[0152] The second queue determination module 1110 is used to determine N second queues according to the N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in the second storage space.

[0153] The data packet determination module 1120 is used to determine a plurality of data packets from the second storage space according to the second storage addresses in the N second queues.

[0154] The image frame determination module 1130 is used to determine a plurality of image frames according to a plurality of data packets.

[0155] The processing module 1140 is used to process multiple image frames using a predetermined algorithm to obtain a processing result.

[0156] According to another embodiment of the present disclosure, the second queue determination module includes: a reading submodule, a storage submodule and a third adding submodule. The reading submodule is used to read multiple target data packets from the first storage space according to the first storage address received in the first queue. The storage submodule is used to store the multiple target data packets in the second storage area. The third adding submodule is used to add the second storage addresses of the multiple target data packets in the second storage area to the second queue.

[0157] Fig.12 It is a structural block diagram of an electronic device used to implement the video data stream processing method and / or image processing method of the embodiments of the present disclosure.

[0158] like Fig.12 As shown, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include an onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0159] In RAM 1203, various programs and data required for the operation of electronic device 1200 are stored. Processor 1201, ROM 1202 and RAM 1203 are connected to each other via bus 1204. Processor 1201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1202 and / or RAM 1203. It should be noted that the program can also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0160] According to an embodiment of the present disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 may further include one or more of the following components connected to the I / O interface 1205: an input portion 1206 including a keyboard, a mouse, etc.; an output portion 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1208 including a hard disk, etc.; and a communication portion 1209 including a network interface card such as a LAN card, a modem, etc. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage portion 1208 as needed.

[0161] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0162] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.

[0163] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the item recommendation method provided by the embodiment of the present disclosure.

[0164] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1201. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0165] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1209, and / or installed from the removable medium 1211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0166] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0167] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0168] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0170] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A video data stream processing method, comprising: Determine N video data streams from N image acquisition devices; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2; Dividing the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to a video data stream; Determine N first queues corresponding to the N data packet sets respectively, each first queue comprising a first storage address in the first storage space of each data packet in the data packet set corresponding to the first queue; and The N first queues are sent.

2. The method according to claim 1, wherein: The step of dividing the plurality of data packets included in the N video data streams into N data packet sets comprises: Determine the video identifier of the video data stream where each data packet is located; Dividing the plurality of data packets according to the video identifier to obtain N initial data packet sets; and The N data packet sets are determined according to the N initial data packet sets.

3. The method according to claim 2, wherein: Determining the N data packet sets according to the N initial data packet sets includes: Sort the data packets in the initial data packet set according to the sequence numbers of the data packets in the initial data packet set to obtain a data packet sequence; and The data packet set is determined according to the data packet sequence.

4. The method according to claim 1, wherein: The data packet set includes at least one data packet subset, each data packet subset includes at least one data packet, and each data packet subset corresponds to an image frame; The determining of N first queues respectively corresponding to the N data packet sets comprises: In response to detecting that the first queue includes a first storage address for a previous data packet, adding a first storage address for a current data packet to the first queue; wherein the current data packet and the previous data packet belong to the same data packet subset; and In response to detecting that the first queue includes a first storage address for a previous subset of data packets, adding a first storage address for a current subset of data packets to the first queue; wherein the current subset of data packets and the previous subset of data packets belong to the same set of data packets.

5. The method according to claim 1, wherein: The data packet set includes at least one data packet subset, each data packet subset corresponds to an image frame; and sending the N first queues includes: In response to detecting that a target storage address in the first queue satisfies a predetermined condition, sending the target storage address; Wherein, the predetermined condition includes: the data packet indicated by the target storage address is used to determine an image frame.

6. The method according to claim 5, wherein: The predetermined condition also includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold.

7. The method according to claim 1, further comprising: In response to the plurality of target data packets in the first storage space being read, the plurality of target data packets are deleted from the first storage space.

8. The method according to claim 1, further comprising: Determine target adjustment parameters according to the scene identifier of the target scene; The target adjustment parameter includes at least one of the following: the target number of the image acquisition device and the target acquisition frequency of the image acquisition device; In response to detecting that the target adjustment parameter includes a target number of the image acquisition devices, adjusting the actual used number of the image acquisition devices according to the target number; as well as In response to detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition device, the actual acquisition frequency of the image acquisition device is adjusted according to the target acquisition frequency.

9. The method according to claim 1, wherein: The data packet includes a network layer abstraction unit.

10. An image processing method, comprising: Determine N second queues according to the N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of the data packets in the second storage space; Determining a plurality of data packets from the second storage space according to the second storage addresses in the N second queues; Determining a plurality of image frames according to the plurality of data packets; as well as The plurality of image frames are processed using a predetermined algorithm to obtain a processing result.

11. The method according to claim 10, wherein: The determining of N second queues according to the N first queues includes: Reading a plurality of target data packets from a first storage space according to the received first storage address in the first queue; storing the plurality of target data packets in a second storage area; Add the second storage addresses of the multiple target data packets in the second storage area to the second queue.

12. A video data stream processing device, comprising: A video data stream determination module, used to determine N video data streams from N image acquisition devices; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2; A division module, used for dividing the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to one video data stream; A first queue determination module, configured to determine N first queues corresponding to the N data packet sets respectively, each first queue comprising a first storage address in a first storage space of each data packet in the data packet set corresponding to the first queue; and A sending module is used to send the N first queues.

13. An image processing device, comprising: A second queue determination module, used to determine N second queues according to the N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of the data packets in the second storage space; a data packet determining module, configured to determine a plurality of data packets from the second storage space according to the second storage addresses in the N second queues; An image frame determination module, used to determine a plurality of image frames according to the plurality of data packets; as well as The processing module is used to process the multiple image frames using a predetermined algorithm to obtain a processing result.

14. A video data stream processing device, comprising a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 9. 15 . An image processing device, comprising a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the method according to claim 10 .

16. An image processing system, comprising: N image acquisition devices, used for acquiring N video data streams; The video data stream processing device according to claim 14; as well as The image processing device as claimed in claim 15.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

18. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images