Video data stream processing method and device, image processing method and device, and storage medium

Abstract

The present disclosure provides a video data stream processing method and device, an image processing method and device, and a storage medium, which can be applied to the technical field of artificial intelligence, and in particular to the technical fields of computer vision, streaming media, etc. The video data stream processing method comprises: determining N video data streams from N image acquisition apparatuses, each video data stream comprising at least one data packet, and N being an integer greater than or equal to 2; dividing a plurality of data packets included in the N video data streams into N data packet sets, each data packet set corresponding to one video data stream; determining N first queues respectively corresponding to the N data packet sets, each first queue comprising a first storage address of each data packet in the data packet set corresponding to the first queue in a first storage space; and sending the N first queues.

Description

Video data stream processing method, image processing method, device and storage medium Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as computer vision and streaming media. Background Art

[0002] Image acquisition devices can be used to capture video stream data, then process the image frames within the video stream data to produce image processing results. For example, multiple video data streams captured by multiple image acquisition devices are processed independently, with each video data stream occupying a separate thread to perform decoding, object detection, and other processing on the video stream data. However, this processing method suffers from low efficiency and low accuracy.

[0003] Summary of the Invention

[0004] The present disclosure provides a video data stream processing method, an image processing method, a video data stream processing apparatus, an image processing apparatus, a video data stream processing device, an image processing device, an image processing system, a storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a video data stream processing method is provided, comprising: determining N video data streams from N image acquisition devices; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2; dividing the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to one video data stream; determining N first queues corresponding to the N data packet sets, each first queue including a first storage address in a first storage space for each data packet in the data packet set corresponding to the first queue; and sending the N first queues.

[0006] For example, dividing multiple data packets included in N video data streams into N data packet sets includes: determining the video identifier of the video data stream in which each data packet is located; dividing the multiple data packets according to the video identifier to obtain N initial data packet sets; and determining N data packet sets based on the N initial data packet sets.

[0007] For example, based on N initial data packet sets, determining N data packet sets includes: sorting the data packets in the initial data packet set according to the serial numbers of the data packets in the initial data packet set to obtain a data packet sequence; and determining the data packet set according to the data packet sequence.

[0008] For example, a data packet set includes at least one data packet subset, each data packet subset includes at least one data packet, and each data packet subset corresponds to an image frame; determining N first queues corresponding to the N data packet sets respectively includes: in response to detecting that the first queue includes a first storage address for a previous data packet, adding the first storage address for the current data packet to the first queue; wherein the current data packet and the previous data packet belong to the same data packet subset; and in response to detecting that the first queue includes a first storage address for a previous data packet subset, adding the first storage address for the current data packet subset to the first queue; wherein the current data packet subset and the previous data packet subset belong to the same data packet set.

[0009] For example, the data packet set includes at least one data packet subset, each data packet subset corresponds to an image frame; sending N first queues includes: in response to detecting that the target storage address in the first queue meets a predetermined condition, sending the target storage address; wherein the predetermined condition includes: the data packet indicated by the target storage address is used to determine an image frame.

[0010] For example, the predetermined condition further includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold.

[0011] For example, the method further includes: in response to the plurality of target data packets in the first storage space being read, deleting the plurality of target data packets from the first storage space.

[0012] For example, it also includes: determining a target adjustment parameter based on a scene identifier of a target scene; the target adjustment parameter includes at least one of the following: a target number of the image acquisition device and a target acquisition frequency of the image acquisition device; in response to detecting that the target adjustment parameter includes the target number of the image acquisition device, adjusting the actual usage number of the image acquisition device according to the target number; and in response to detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition device, adjusting the actual acquisition frequency of the image acquisition device according to the target acquisition frequency.

[0013] For example, a data packet includes a network layer abstraction unit.

[0014] According to another aspect of the present disclosure, an image processing method is provided, including: determining N second queues based on N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in the second storage space; determining multiple data packets from the second storage space based on the second storage addresses in the N second queues; determining multiple image frames based on the multiple data packets; and processing the multiple image frames using a predetermined algorithm to obtain processing results.

[0015] For example, based on N first queues, determining N second queues includes: reading multiple target data packets from the first storage space based on the first storage address received in the first queue; storing the multiple target data packets in the second storage area; and adding the second storage addresses of the multiple target data packets in the second storage area to the second queue.

[0016] According to another aspect of the present disclosure, a video data stream processing device is provided, comprising: a video data stream determination module, a division module, a first queue determination module, and a sending module. The video data stream determination module is used to determine N video data streams from N image acquisition devices; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2. The division module is used to divide the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to a video data stream. The first queue determination module is used to determine N first queues corresponding to the N data packet sets, each first queue including the first storage address of each data packet in the data packet set corresponding to the first queue in the first storage space. The sending module is used to send the N first queues.

[0017] According to another aspect of the present disclosure, an image processing device is provided, comprising: a second queue determination module, a data packet determination module, an image frame determination module, and a processing module. The second queue determination module is configured to determine N second queues based on N first queues, where N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in a second storage space. The data packet determination module is configured to determine multiple data packets from the second storage space based on the second storage addresses in the N second queues. The image frame determination module is configured to determine multiple image frames based on the multiple data packets. The processing module is configured to process the multiple image frames using a predetermined algorithm to obtain processing results.

[0018] According to another aspect of the present disclosure, a video data stream processing device is provided, including a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the above-mentioned video data stream processing method.

[0019] According to another aspect of the present disclosure, an image processing device is provided, including a memory and a processor, wherein instructions executable by the processor are stored in the memory, and when the instructions are executed by the processor, the processor executes the above image processing method.

[0020] According to another aspect of the present disclosure, an image processing system is provided, comprising: N image acquisition devices for acquiring N video data streams, the above-mentioned video data stream processing device, and the above-mentioned image processing device.

[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0022] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0025] FIG1 is a schematic flow chart of a method for processing a video data stream according to an embodiment of the present disclosure;

[0026] FIG2 is a schematic flow chart of an image processing method according to an embodiment of the present disclosure;

[0027] FIG3 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

[0028] FIG4 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

[0029] FIG5 is a schematic diagram of the structure of multiple data packets according to an embodiment of the present disclosure;

[0030] FIG6 is a schematic diagram of a data structure of a data packet according to an embodiment of the present disclosure;

[0031] FIG7 is a schematic diagram of data packet extraction according to an embodiment of the present disclosure;

[0032] FIG8 is a schematic diagram of a processing process of an image acquisition device according to an embodiment of the present disclosure;

[0033] FIG9 is a schematic structural diagram of an image processing device according to another embodiment of the present disclosure;

[0034] FIG10 is a schematic structural block diagram of a video data stream processing apparatus according to an embodiment of the present disclosure;

[0035] FIG11 is a schematic structural block diagram of an image processing apparatus according to an embodiment of the present disclosure; and

[0036] FIG12 is a structural block diagram of an electronic device for implementing the video data stream processing method and / or image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0038] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0039] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0040] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0041] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0042] The present disclosure provides a video data stream processing method and an image processing method. The video data stream processing method can be executed by a video data stream processing device, which is a source. The image processing method can be executed by an image processing device, which is a sink.

[0043] During processing, the video data stream processing device may obtain the video data stream captured by the image acquisition device, determine N first queues based on the video data stream, and send the N first queues to the image processing device. The image processing device may then determine image frames based on the N first queues and process the image frames to obtain processing results.

[0044] The technical solution provided by the embodiments of the present disclosure achieves efficient access to multiple streaming media by performing splicing, extraction, and other processing on data packets, determines multiple image frames based on multiple video stream data collected by multiple image acquisition devices, and then batch processes the multiple image frames, thereby solving the problem of inefficiency in multi-threaded processing of multiple streaming media, as well as the problem of decreased overall accuracy and increased false alarm rate of computer vision applications caused by this inefficiency, and can also reduce equipment load, reduce costs, and improve equipment utilization.

[0045] The technical solution provided in the embodiments of the present disclosure is applicable to streaming media computer vision applications. In the actual deployment process, the solution provided in the present disclosure is applicable to, for example, the server side (such as x86 server), box (such as ARM box), Android, etc., and can be deployed on terminal devices such as televisions and mobile phones.

[0046] The video data stream processing method and the image processing method are described below.

[0047] FIG1 is a schematic flowchart of a method for processing a video data stream according to an embodiment of the present disclosure.

[0048] As shown in FIG1 , the video data stream processing method 100 may be executed by a video data stream processing device. The method 100 includes operations S110 to S140 .

[0049] In operation S110 , N video data streams from N image acquisition devices are determined; each video data stream includes at least one data packet, and N is an integer greater than or equal to 2.

[0050] For example, the image acquisition device may include a camera, such as a camera installed in a park, a shopping mall, or a camera installed on a mobile object such as an autonomous vehicle. N may be 4, 16, or the like.

[0051] For example, the image acquisition device may use a video data stream, and N image acquisition devices correspond one to one with N video data streams. For example, the first image acquisition device Cam1 acquires the first video data stream Video1, and the second image acquisition device Cam2 acquires the second video data stream Video2.

[0052] It should be noted that, in the actual processing process, the video data stream collected by the image acquisition device may be an H.264 code stream, which may be RTSP encapsulated and then the data packets in the video data stream may be obtained by decompressing the RTSP and H.264 code stream.

[0053] In operation S120, a plurality of data packets included in the N video data streams are divided into N data packet sets, each data packet set corresponding to one video data stream.

[0054] It should be noted that the data packets in the N data streams enter the first storage space in chronological order, so the multiple data packets in the first storage space are out of order. The multiple data packets in the first storage space can be divided to obtain N data packet sets.

[0055] For example, multiple data packets in a first video data stream from a first image acquisition device Cam1 are divided into a data packet set A1, and multiple data packets in a video data stream from a second image acquisition device Cam2 are divided into a second data packet set A2.

[0056] It should be noted that the video data stream captured by an image acquisition device can determine a data packet set. A data packet set can include at least one data packet subset, and a data packet subset can include at least one data packet. The data packets in a data packet subset can be used to determine a complete image frame. For example, the data packet set determined by the video data stream Video1 can include data packets package1, package2, package3, package4, package5, package6, and package7. The data packet set can be a first data packet subset and a second data packet subset. The first data packet subset can include data packets package1, package2, and package3. The image processing device can use the first data packet set to determine an image frame, and the second data packet subset can include package4, package5, package6, and package7. The image processing device can use the second data packet set to determine another image frame. The above-mentioned data packets can be, for example, RTP protocol data packets, such as H264 compressed multiplexed packets.

[0057] It should be noted that during the transmission of a video data stream, if the data that can identify a complete image frame is recorded in a single data packet, the data volume of the data packet is large. During the actual transmission process, if the data packet transmission fails due to network anomalies or other reasons, the entire data that can identify a complete image frame will need to be retransmitted. However, dividing the data that can identify a complete image frame into multiple data packets (these multiple data packets form a data packet subset) allows the retransmission of a data packet after a transmission failure, without having to transmit the entire data packet subset, thereby improving transmission efficiency.

[0058] In operation S130 , N first queues corresponding to the N data packet sets are determined, each first queue including a first storage address in the first storage space of each data packet in the data packet set corresponding to the first queue.

[0059] For example, the first storage space may be a memory of a video data stream processing device, and the storage address of the data packet in the first storage space is referred to as a first storage address.

[0060] For example, N data packet sets correspond one-to-one to N first queues, for example, data packet set A1 corresponds to the first queue Q1, and data packet set A2 corresponds to the first queue Q2. Therefore, the first storage address of the data packet in data packet set A1 can be added to the first queue Q1, and the first storage address of the data packet in data packet set A2 can be added to the first queue Q2.

[0061] In operation S140 , N first queues are transmitted.

[0062] For example, N first queues are sent to an image processing device. The transmission mode of the first queues may be a wired transmission mode such as PCIE, or a wireless transmission mode such as WiFi.

[0063] The technical solution provided by the disclosed embodiments determines N first queues based on N video data streams. Each first queue includes the first storage addresses of multiple data packets from an image acquisition device. During processing by the image processing device, multiple image frames can be determined based on the first queues, and batch processing can be performed on the multiple image frames. This maximizes the use of the image processing device's computing resources, improving computational efficiency and accuracy.

[0064] In addition, the information in the first queue is the first storage address of the data packet, not the data packet itself. Therefore, during the data transmission process, the data packet only needs to be copied once, and there is no need to copy the data table to the first queue, thereby reducing the amount of data transmission, improving data transmission efficiency, and reducing the requirements for device hardware.

[0065] According to another embodiment of the present disclosure, the above-mentioned process of dividing the multiple data packets included in N video data streams into N data packet sets may include: determining the video identifier of the video data stream in which each data packet is located, and then dividing the multiple data packets according to the video identifier to obtain N initial data packet sets, and then determining N data packet sets based on the N initial data packet sets.

[0066] For example, a data packet may include a video identifier, and multiple data packets may be divided based on the video identifier to obtain multiple initial data packet sets, each of which corresponds to the same video identifier. For example, the video identifiers of the multiple data packets in the first initial data packet set are all media1, while the video identifiers of the multiple data packets in the second initial data packet set are all media2. This embodiment divides the data packet sets based on the video identifiers, so that a first queue stores the first storage address of a data packet from a single image acquisition device, thereby ensuring a one-to-one correspondence between N initial data packet sets and N video data streams.

[0067] For example, N initial data packet sets may be determined as N data packet sets, and the data packets in the data packet sets may be randomly ordered.

[0068] For another example, the data packets in the initial data packet set may be sorted according to the serial numbers of the data packets in the initial data packet set to obtain a data packet sequence, and then the data packet set may be determined according to the data packet sequence.

[0069] For example, when transmitting data packets based on the RTP protocol or TCP protocol, each data packet has a serial number, which is also called a sequence number. The serial number can be a long integer data, and multiple serial numbers can be continuous. The data packets in the data packet set are arranged in sequence according to the serial number.

[0070] In this embodiment, the data packets in the data packet set are arranged sequentially according to their serial numbers. Compared to a random arrangement, the technical solution provided by this embodiment ensures that, after determining the first queue based on the data packet set, the first storage addresses in the first queue are arranged sequentially according to the serial numbers of the data packets, thereby ensuring that image frames can be subsequently determined based on the ordered data packets. Furthermore, during processing, the image processing device needs to decode the data packets in order to avoid decoding errors. Since the data packets in the data packet set in this embodiment are arranged in order of their serial numbers, the image processing device can decode the data packets sequentially, eliminating the need to re-determine the decoding order based on random data packets. This improves the processing efficiency of the image processing device and saves performance.

[0071] According to another embodiment of the present disclosure, when determining N first queues corresponding to N data packet sets, for different data packet subsets within the same data packet set, after detecting that the first queue includes the first storage address for the previous data packet subset, the first storage address for the current data packet subset can be added to the first queue. In this way, the first storage addresses of multiple data packets within the same data packet subset can be sequentially added to the first queue, thereby ensuring the order of the data packets.

[0072] According to another embodiment of the present disclosure, in the process of determining N first queues corresponding to N data packet sets, for different data packets in the same data packet subset, where the same data packet subset includes the current data packet and at least one previous data packet, after detecting that the first queue includes the first storage address for the previous data packet, the first storage address for the current data packet can be added to the first queue. In this way, the first storage addresses of multiple data packet subsets in the same data packet set can be added to the first queue in sequence, thereby ensuring the orderliness of the data packet subsets.

[0073] According to another embodiment of the present disclosure, in the process of sending N first queues, the target storage address may be sent after detecting that the target storage address in the first queue meets a predetermined condition.

[0074] For example, the predetermined conditions include: the data packet indicated by the target storage address is used to determine an image frame, the number of target storage addresses can be multiple, and the multiple data packets stored in the multiple target storage addresses can determine a complete image frame, thereby avoiding the image processing device being unable to obtain a complete image frame based on the data packet during subsequent processing, thereby ensuring the accuracy of the image frame processing results.

[0075] For another example, the predetermined condition includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold. The threshold can be an integer greater than or equal to 2, for example, any value between 8 and 16. The threshold can be used to control the number of image frames batch processed by the image processing device, thereby avoiding wasting computing resources of the image processing device due to a small number of image frames and ensuring data processing efficiency.

[0076] According to another embodiment of the present disclosure, the video data stream processing method may further include the following operations: determining a target adjustment parameter according to a scene identifier of a target scene, and then adjusting usage of N image acquisition devices according to the target adjustment parameter.

[0077] For example, a scenario can be related to the data processing capabilities of a device. A machine may include at least one of a video data stream processing device and an image processing device. The device's data processing capabilities can be determined based on hardware information such as the number of processor cores. The correspondence between device information and scenarios can be preconfigured so that, after determining the device information of the currently used device, the scenario identifier can be determined based on this correspondence.

[0078] For example, the correspondence between scenes and adjustment parameters may be configured in advance according to actual needs, and then after obtaining the scene identifier of the target scene, the adjustment parameters corresponding to the scene identifier are searched based on the scene identifier and the correspondence as the target adjustment parameters.

[0079] The target adjustment parameter may include a target number of image acquisition devices. For example, some devices may have weak data processing capabilities and may only be able to process video data streams collected by five image acquisition devices, while other devices may have stronger data processing capabilities and may be able to process video data streams collected by ten image acquisition devices. Therefore, after detecting that the target adjustment parameter includes the target number of image acquisition devices, the actual number of image acquisition devices in use may be adjusted based on the target number, for example, so that the adjusted actual number of image acquisition devices in use is consistent with the target number, thereby matching the number of video data streams with the processing capabilities of the devices.

[0080] The target adjustment parameter may include a target acquisition frequency of the image acquisition device. For example, some devices have weak data processing capabilities and can process video data streams with an fps (frames per second) of 12, while other devices have strong data processing capabilities and can process video data streams with an fps of 25. Therefore, after detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition device, the actual acquisition frequency of the image acquisition device can be adjusted based on the target acquisition frequency, for example, so that the actual acquisition frequency of the image acquisition device after adjustment is consistent with the target frequency, thereby matching the acquisition frequency of the video data stream with the processing capability of the device.

[0081] FIG2 is a schematic flowchart of an image processing method according to an embodiment of the present disclosure.

[0082] As shown in FIG. 2 , the image processing method 200 may be executed by an image processing device. The method 200 may include operations S210 to S240 .

[0083] In operation S210 , N second queues are determined according to the N first queues, where N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in the second storage space.

[0084] For example, the video data stream processing device may adopt the above-mentioned video data stream processing method to obtain N first queues, and then send the N first queues to the image processing device.

[0085] In one example, the first storage address in the first queue indicates the address of the data packet in the first storage space, and the second storage address in the second queue indicates the address of the data packet in the second storage space. Based on the first storage address received in the first queue, the image processing device can read multiple target data packets from the first storage space, store the multiple target data packets in the second storage area, and then add the second storage addresses of the multiple target data packets in the second storage area to the second queue.

[0086] In another example, the image processing device and the video stream processing device share the same storage space. In this case, the N first queues can be determined as N second queues.

[0087] In operation S220 , a plurality of data packets are determined from the second storage space according to the second storage addresses in the N second queues.

[0088] For example, for each second queue, a data packet at the address may be read from the second storage space based on the second storage address in the second queue.

[0089] In operation S230, a plurality of image frames are determined based on the plurality of data packets.

[0090] For example, the read data packets are decoded and processed, and then the data packets in the data packet subset corresponding to an image frame are used to determine an image frame. By processing multiple data packet subsets in this way, multiple image frames can be obtained, and the multiple image frames constitute a batch of data.

[0091] In operation S240 , the plurality of image frames are processed using a predetermined algorithm to obtain a processing result.

[0092] For example, a deep learning model can be used as the predetermined algorithm. The deep learning model includes, for example, YOLO. Multiple images can be input into the YOLO model, and the model outputs a detection box, which can be used as a processing result.

[0093] The technical solution provided in this embodiment determines N first queues based on N video data streams. Each first queue includes the first storage addresses of multiple data packets from a single image acquisition device. During processing by the image processing device, multiple image frames can be determined based on the first queues, and batch processing can be performed on the multiple image frames. This maximizes the use of the image processing device's computing resources, improving computational efficiency and accuracy.

[0094] In some embodiments, after the image processing device reads the plurality of target data packets from the first storage space, the video data stream processing device deletes the plurality of read target data packets from the first storage space so that the first storage space receives new data packets.

[0095] FIG3 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure.

[0096] As shown in FIG. 3 , in this embodiment, the platform includes image acquisition devices 311 , 312 , 313 , a video data stream processing device 320 , and an image processing device 330 .

[0097] The image acquisition devices 311 , 312 , 313 are used to acquire video data streams. The number of the image acquisition devices 311 , 312 , 313 may be N, where N is an integer greater than or equal to 2. Three are shown in the figure.

[0098] The video data stream processing device 320 can obtain N video data streams from N image acquisition devices 311, 312, and 313, each video data stream including at least one data packet. The video data stream processing device 320 can divide the multiple data packets in the N video data streams into N data sets and determine N first queues. The N first queues can then be sent to the image processing device 330 via the connector 340 or other means. The connector 340 can be a PCIe (Peripheral Component Interconnect Express) connector.

[0099] The image processing device 330 may determine N second queues based on the N first queues, and then subsequent services may consume the second storage addresses in the second queues as consumers, thereby implementing the process of determining and processing image frames.

[0100] FIG4 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure.

[0101] As shown in Figure 4 , in this embodiment, the platform includes an image acquisition device 410, a video data stream processing device 420, and an image processing device 430. Data is transmitted between the video data stream processing device 420 and the image processing device 430 via a connector 440. This embodiment primarily describes the data processing processes of the video data stream processing device 420 and the image processing device 430 from a software perspective.

[0102] The number of the image acquisition devices 410 is N, which may include, for example, image acquisition devices 411 , 412 , and 413 . The N image acquisition devices 410 may transmit N video data streams to the video data stream processing device 420 .

[0103] The video data stream processing device 420 includes multiple functional modules, such as an identifier generation module 421, a data packet combination module 422, a data packet splicing module 423, a data packet extraction module 424, and a first queue module 425. The functional modules in the video data stream processing device 420 are described below.

[0104] Identifier generation module 421 obtains a video data stream from image acquisition device 410. The video data stream includes multiple data packets. Identifier generation module 421 generates a video identifier for each data packet. This video identifier identifies the video data stream to which the data packet belongs and also identifies the image acquisition device 410 that sent the data packet. The video identifier can be added to an extension field in the data packet. Next, identifier generation module 421 sends the video identifier and the data packet to data packet assembly module 422. It should be noted that identifier generation module 421 can obtain the video data stream from image acquisition device 410 asynchronously. The main reason for this asynchronous operation is network congestion. For example, if a video data stream includes multiple data packets and the second data packet is delayed due to network congestion, the third data packet may be sent before the second data packet. In this case, video data stream processing device 420 asynchronously receives the second data packet. By using an asynchronous method to process the video data stream, image processing device 430 does not need to wait for data packets that were temporarily unsuccessfully sent due to network congestion, thereby improving data packet transmission efficiency.

[0105] The data packet combining module 422 stores the data packet in a first storage space, for example, in a memory of the video data stream processing device 420 .

[0106] The data packet splicing module 423 splices the multiple data packets in the first storage space into a time sequence stream with a video identifier. The data packets can be arranged according to the time sequence of the data packets received in the first storage space.

[0107] During the actual splicing process, the data packet set can form an N-dimensional array (ndarray) structure. Multiple data packets corresponding to the same video identifier can be first divided into the same initial data packet set, and then the multiple data packets in the initial data packet set are sorted according to the serial number. The sorted initial data packet set is the data packet set mentioned above. Next, a structural diagram of the N-dimensional array can be formed based on the order of the data packet set and the order of each data packet in the same data packet set. This structure facilitates the storage and use of data packets. Since the data of the structure comes from N image acquisition devices 410, and a data packet subset after splicing includes multiple (for example, 3) data packets, the N-dimensional array of these data packets can be easily operated on and retrieved in rows and columns. In other embodiments, the data packet set can also form a linked list structure. It can be seen that if the data packet is a NALU (Network Abstract Layer Unit), the number of NALUs contained in the spliced ​​data structure will increase. The structure of the NALU data packet is described below and will not be repeated here.

[0108] It should be noted that if there is only one video data stream, the data packets in the first storage space correspond only to that one video data stream. However, if there are at least two video data streams, the data packets in the at least two video data streams enter the first storage space in chronological order, so the multiple data packets in the first storage space are out of order. For example, the order of the multiple video data packets in the first storage space may be: the first data packet in the first video data packet, the first data packet in the second video data packet, the second data packet in the first video data packet, the first data packet in the third video data packet, the third data packet in the first video data packet, and so on.

[0109] In this embodiment, the data packet concatenation module 423 groups and sorts the multiple data packets in the first storage space, resulting in N data packet sets. The data packets within each data packet set are arranged in sequence, and the first queue reflects the grouping and sorting of the data packets. In subsequent processing, the image processing device can sequentially process the data packets based on the first queue to obtain multiple image frames, and then perform batch processing on the multiple image frames, thereby utilizing more computing resources of the image processing device and improving computational efficiency and accuracy.

[0110] The data packet extraction module 424 can extract the first storage address from the structure formed by the data packet. For example, the data packet includes multiple fields, where the data field represents the specific data content in the data packet. The first storage address can include the starting position and ending position of the data field, or the starting position and offset.

[0111] The first queue module 425 maintains N first queues. For a corresponding first queue and data packet set, the first storage address of a data packet in the data packet set can be added to the first queue.

[0112] The image processing device 430 includes multiple functional modules, such as a second queue module 431, a depacketizing module 432, and a processing module 433. The functional modules in the image processing device 430 are described below.

[0113] The second queue module 431 maintains N second queues. The data structure of the first queue and the data structure of the second queue can be the same. The difference between the first queue and the second queue is that the first storage address in the first queue indicates the address of the data packet in the first storage space, such as the CPU memory. The second storage address in the second queue indicates the address of the data packet in the second storage space, such as the GPU video memory. Both the first queue and the second queue can be shared CV (computer vision) frame queues between processes.

[0114] The unpacking module 432 can obtain the second storage address from the second queue, then read the data packet in the second storage space based on the second storage address, and then perform a predetermined decoding process, thereby processing the single data packet subset into an image frame, thereby obtaining multiple image frames (Frame), which can be used as a batch of (Batch) data, and the batch of data can be used for AI business processing, alarm generation, and other functions. It should be noted that the process of processing the single data packet subset into an image frame may include decoding, and this embodiment does not limit the decoding method.

[0115] In some embodiments, multiple image frames within a batch of data can be associated. For example, the batch of input data required for model processing for a facial recognition service may come from multiple image acquisition devices. When performing cross-border head tracking, the same object may appear in the field of view of certain image acquisition devices. In this case, it is not necessary to use all data at all times as input data for the model; only the data packets for the object collected by these specific image acquisition devices can be considered as a batch of data. In other embodiments, the multiple image frames within a batch of data can also be unrelated.

[0116] Processing module 433 is used to process multiple image frames to obtain processing results. Processing module 433 may, for example, include a CV pre-processing (CV_RESIZE) module and a deep learning model. The deep learning model may be, for example, a YOLO model, which may include, for example, a convolutional layer. When processing multiple image frames in a batch of data, processing module 433 does not impose any restrictions on the order of the multiple image frames. The multiple image frames may be processed sequentially in a chronological order or in a random order.

[0117] It should be noted that the deep learning model can be determined according to actual needs, and this embodiment does not limit the deep learning model. The above-mentioned YOLO model can be applied to scenarios for identifying objects in places such as parks and the like, and the objects may include users, vehicles, etc. The image processing device 430 can determine the source of the data packet based on the video identifier of the video to which the data packet belongs, thereby performing data analysis in combination with the actual position of the image acquisition device 410. For example, based on the identified users, the recognition accuracy of multiple people at various angles can be confirmed. Based on the identified vehicles, vehicle tracking can be performed. For example, the data packets of multiple image acquisition devices 410 can be spliced ​​into a group of image frames, which can improve speed performance and analyze vehicles from multiple angles, thereby improving the accuracy of vehicle recognition.

[0118] Compared to an approach that employs only one image acquisition device 410, this embodiment employs N image acquisition devices 410. The video data stream processing device 420 can output N first queues, and the image processing device 430 can then process a batch of image frames. It should be noted that the processing module 433 can utilize a parallel processing approach during processing. The processing model can process one image frame in 20 ms, and the time required to process eight image frames is only slightly more than 20 ms. Therefore, for a predetermined number of image frames, processing a larger number of image frames in a single process utilizes more computing resources of the image processing device 430, thereby improving processing efficiency for the predetermined number of image frames. It can be seen that this embodiment, by increasing the batch size, provides the processing module 433 with more image frames during a single code execution cycle. This allows for batch processing of image frames, reducing the computational complexity and storage requirements for packet encapsulation and unpacking at each stage of data transmission, thereby improving processing efficiency.

[0119] FIG5 is a schematic diagram of the structure of multiple data packets according to an embodiment of the present disclosure.

[0120] For example, the data packet splicing module mentioned above splices multiple data packets in the first storage space into a time sequence stream with a video identifier. It can be understood that the various data packet subsets in the data packet set in the first storage space are arranged in time sequence, and the various data packets in the same data packet subset are arranged in serial numbers. The above-mentioned time sequence stream can represent: a data packet set after multiple data packets are arranged in time sequence.

[0121] Figure 5 shows two data packet sets, where each small square represents a data packet. One data packet set includes multiple data packet subsets. For example, one data packet subset includes data packets 1-1, 1-2, and 1-3, while the other data packet subset includes data packets 1-4, 1-5, and 1-6. The other data packet set may also include multiple data packet subsets, each data packet subset including at least one data packet. For example, one data packet subset includes data packet 2-1.

[0122] FIG6 is a schematic diagram of a data structure of a data packet according to an embodiment of the present disclosure.

[0123] As shown in Figure 6, in some embodiments, the data packet can be a NALU. The data structure of the NALU is shown in Figure 6. For example, the first bit is 0, the fourth to eighth bits represent the type type, and the second eight bits are specific NALU data. A shaded area in Figure 6 represents specific NALU data 601. Following the specific NALU data 601 is an extension field 602, to which information such as a video identifier and a serial number can be added. Information such as a video identifier and a serial number can also be stored in other storage areas outside the data packet, and a mapping relationship between the above-mentioned information and the data packet can be established based on the actual storage situation. Based on the mapping relationship, the video identifier, serial number, and first storage address corresponding to each data packet can be queried.

[0124] FIG7 is a schematic diagram of data packet extraction according to an embodiment of the present disclosure.

[0125] For example, the first storage address can be extracted from the structure formed by the data packet 7221. For example, a data packet has multiple fields, wherein the Data field represents the specific data content in the data packet, and the first storage address can include the starting position and the ending position of the Data field, or the starting position and the offset. The first storage address is then added to the first queue 7251. For example, the first storage address of a data packet in another data packet set can be added to another first queue 7252. In addition, the data packet can also have an extension field, which can be used to store a video identifier and a serial number.

[0126] FIG8 is a schematic diagram of a processing procedure of an image acquisition device according to an embodiment of the present disclosure.

[0127] During actual processing, for example, the first process 901 of a video data stream processing device can add data packets 1-20 to the first queue 805. Of these data packets, data packets 1-9 form the first data packet subset, data packets 10-19 form the second data packet subset, and data packets 20-30 form the third data packet subset. The image processing device has a large number of GPU cores and can run a large number of processes in parallel. For example, the number of cores can be the same as the number of parallel threads, and a thread can include multiple processes. As shown in Figure 8, the second process 802 can be used to pull the first data packet subset from the second queue 806, the third process 803 can be used to pull the second data packet subset from the second queue 806, and the fourth process 804 can be used to pull the third data packet subset from the second queue 806. The second process 802, the third process 803, and the fourth process 804 can be processed in parallel.

[0128] FIG9 is a schematic structural diagram of an image processing device according to another embodiment of the present disclosure.

[0129] As shown in Figure 9, in this embodiment, the platform includes image acquisition devices 911, 912, 913, a video data stream processing device 920, an image processing device 930, a PCIe bus 940, and a network card 950. This embodiment mainly describes the platform from a hardware perspective.

[0130] N image acquisition devices are used to capture video data streams, such as RTSP streams, which include RTP packets, each of which includes H.264 NALUs. The video data streams are first transmitted by the image acquisition devices to the network card 950 and then transmitted via the PCIe bus 940 to the processor 921 in the video data stream processing device 920.

[0131] The video data stream processing device 920 includes at least one processor 921, each of which may include at least one computing unit 9211. If a processor 921 includes four computing units 9211, then the number of cores of the processor 921 is four. The computing unit 9211 may process a video data stream, which includes multiple data packets, each of which may be a NALU. The video data stream processing device 920 processes multiple data packets to obtain N first queues. Next, the multiple data packets in the N first queues and / or the N video data streams are transmitted to the image processing device 930 via the PCIe bus 940. The image processing device 930 processes the multiple data packets to determine image frames and processing results for the image frames.

[0132] It should be noted that Figure 4 mainly illustrates the data processing process of the video data stream processing device from a software perspective, and Figure 9 mainly illustrates the video data stream processing device from a hardware perspective. The process of processing data packets by the operation unit 9211 in Figure 9 can refer to the data processing process of the identification generation module 421, the data packet combination module 422, the data packet splicing module 423, the data packet extraction module 424 and the first queue module 425 in Figure 4, which will not be repeated here in this embodiment.

[0133] It should be noted that Figure 9 shows four processors 921 and four computing units 9211, wherein the processor 921 including the computing unit 9211 below is used to illustrate one of the processors 921 above, and the computing unit 9221 below is used to illustrate the first computing unit 9221 above.

[0134] FIG10 is a schematic structural block diagram of a video data stream processing apparatus according to an embodiment of the present disclosure.

[0135] As shown in FIG. 10 , the video data stream processing apparatus 1000 may include a video data stream determining module 1010 , a dividing module 1020 , a first queue determining module 1030 , and a sending module 1040 .

[0136] The video data stream determination module 1010 is configured to determine N video data streams from N image acquisition devices, where each video data stream includes at least one data packet, and N is an integer greater than or equal to 2. For example, the video data stream determination module 1010 can be configured to implement a portion of the functionality of the identifier generation module 421 described above.

[0137] The division module 1020 is used to divide the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to a video data stream. For example, the implementation of the division module 1020 can refer to the data packet combination module 422, the data packet splicing module 423 and the data packet extraction module 424 above.

[0138] First queue determination module 1030 is configured to determine N first queues corresponding to the N data packet sets, each of which includes a first storage address in the first storage space for each data packet in the data packet set corresponding to the first queue. The implementation of first queue determination module 1030 may refer to first queue module 425 described above.

[0139] The sending module 1040 is configured to send N first queues.

[0140] According to another embodiment of the present disclosure, a partitioning module includes an identifier determination submodule, a partitioning submodule, and a data packet set determination submodule. The identifier determination submodule is configured to determine the video identifier of the video data stream in which each data packet resides. The partitioning submodule is configured to partition multiple data packets based on the video identifier to obtain N initial data packet sets. The data packet set determination submodule is configured to determine N data packet sets based on the N initial data packet sets.

[0141] According to another embodiment of the present disclosure, the data packet set determination submodule includes: a sequence determination unit and a data packet set determination unit. The sequence determination unit is configured to sort the data packets in the initial data packet set according to their sequence numbers to obtain a data packet sequence; the data packet set determination unit is configured to determine the data packet set based on the data packet sequence. For example, the implementation of the sequence determination unit and the data packet set determination unit may refer to the data packet splicing module 423 in Figure 4.

[0142] According to another embodiment of the present disclosure, a data packet set includes at least one data packet subset, each data packet subset includes at least one data packet, and each data packet subset corresponds to an image frame. The first queue determination module includes a first adding submodule and a second adding submodule. The first adding submodule is configured to, in response to detecting that the first queue includes the first storage address for a previous data packet, add the first storage address for a current data packet to the first queue; wherein the current data packet and the previous data packet belong to the same data packet subset. In this manner, the first storage addresses of multiple data packets in the same data packet subset can be sequentially added to the first queue, thereby ensuring the order of the data packet subsets, that is, the order of the image frames. The second adding submodule is configured to, in response to detecting that the first queue includes the first storage address for a previous data packet subset, add the first storage address for the current data packet subset to the first queue; wherein the current data packet subset and the previous data packet subset belong to the same data packet set. In this manner, the first storage addresses of multiple data packet subsets in the same data packet set can be sequentially added to the first queue, thereby ensuring the order of the data packet subsets, so that subsequent image processing equipment can process the data packets as needed.

[0143] According to another embodiment of the present disclosure, the data packet set includes at least one data packet subset, each data packet subset corresponds to an image frame; the sending module includes: a sending submodule, configured to send a target storage address in response to detecting that a target storage address in a first queue meets a predetermined condition; wherein the number of target storage addresses may be at least one, and the predetermined condition includes: at least one data packet indicated by the at least one target storage address is used to determine a complete image frame.

[0144] According to another embodiment of the present disclosure, the predetermined condition further includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold. For example, the serial number of a data packet can indicate whether the data packet is the last data packet required for decoding to obtain an image frame. Therefore, the serial number can be used to determine the number of complete image frames corresponding to multiple data packets in the N data packet sets.

[0145] For example, the threshold value may be pre-configured according to the size of the image frame. For example, if the preset data volume for a single transmission is 20M and the data volume for decoding an image frame is approximately 2M, the threshold value may be set to 10.

[0146] For example, the threshold may be 1, which indicates real-time transmission, that is, when a data packet for decoding a complete image frame is received, the first queue having the first storage address storing the data packets may be sent.

[0147] For another example, the threshold may be greater than 1, which indicates batch transmission, that is, the first queue is transmitted only after the number of complete image frames corresponding to the data packets reaches the threshold.

[0148] According to another embodiment of the present disclosure, the system further includes: a deletion module configured to delete the plurality of target data packets from the first storage space in response to the plurality of target data packets being read from the first storage space. It should be noted that in this embodiment, the data packets read from the first storage space are referred to as target data packets.

[0149] According to another embodiment of the present disclosure, the system further includes: a parameter determination module, a quantity adjustment module, and a frequency module. The parameter determination module is configured to determine a target adjustment parameter based on a scene identifier of a target scene; the target adjustment parameter includes at least one of the following: a target number of image acquisition devices and a target acquisition frequency of the image acquisition devices; the quantity adjustment module is configured to, in response to detecting that the target adjustment parameter includes the target number of image acquisition devices, adjust the actual number of image acquisition devices in use based on the target number; and the frequency module is configured to, in response to detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition devices, adjust the actual acquisition frequency of the image acquisition devices based on the target acquisition frequency.

[0150] According to another embodiment of the present disclosure, the data packet includes a network layer abstraction unit.

[0151] FIG11 is a schematic structural block diagram of an image processing apparatus according to an embodiment of the present disclosure; and

[0152] As shown in FIG. 11 , the image processing apparatus 1100 may include a second queue determination module 1110 , a data packet determination module 1120 , an image frame determination module 1130 , and a processing module 1140 .

[0153] The second queue determination module 1110 is configured to determine N second queues according to the N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of data packets in the second storage space.

[0154] The data packet determination module 1120 is configured to determine a plurality of data packets from the second storage space according to the second storage addresses in the N second queues.

[0155] The image frame determination module 1130 is configured to determine a plurality of image frames according to a plurality of data packets.

[0156] The processing module 1140 is configured to process multiple image frames using a predetermined algorithm to obtain a processing result.

[0157] According to another embodiment of the present disclosure, the second queue determination module includes a reading submodule, a storage submodule, and a third adding submodule. The reading submodule is configured to read multiple target data packets from a first storage space based on a first storage address received in the first queue. The storage submodule is configured to store the multiple target data packets in a second storage area. The third adding submodule is configured to add the multiple target data packets at the second storage addresses in the second storage area to the second queue.

[0158] FIG12 is a structural block diagram of an electronic device for implementing the video data stream processing method and / or image processing method according to an embodiment of the present disclosure.

[0159] As shown in Figure 12, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1208 into a random access memory (RAM) 1203. The processor 1201 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include an onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0160] Various programs and data required for the operation of the electronic device 1200 are stored in the RAM 1203. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. The processor 1201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1202 and / or the RAM 1203. It should be noted that the programs may also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0161] According to an embodiment of the present disclosure, electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to bus 1204. Electronic device 1200 may further include one or more of the following components connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a LAN card or a modem. Communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in drive 1210 as needed, so that computer programs read from the removable media can be installed into storage section 1208 as needed.

[0162] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0163] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.

[0164] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the item recommendation method provided by the embodiments of the present disclosure.

[0165] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 1201 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0166] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1209, and / or installed from the removable medium 1211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0167] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209 and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0168] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0171] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A video data stream processing method, comprising: Determine N video data streams from N image acquisition devices; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2; Dividing the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to a video data stream; Determine N first queues corresponding to the N data packet sets respectively, each first queue comprising a first storage address in the first storage space of each data packet in the data packet set corresponding to the first queue; and The N first queues are sent.

2. The method according to claim 1, wherein: The step of dividing the plurality of data packets included in the N video data streams into N data packet sets comprises: Determine the video identifier of the video data stream where each data packet is located; Dividing the plurality of data packets according to the video identifier to obtain N initial data packet sets; and The N data packet sets are determined according to the N initial data packet sets.

3. The method according to claim 2, wherein: Determining the N data packet sets according to the N initial data packet sets includes: Sort the data packets in the initial data packet set according to the sequence numbers of the data packets in the initial data packet set to obtain a data packet sequence; and The data packet set is determined according to the data packet sequence.

4. The method according to claim 1, wherein: The data packet set includes at least one data packet subset, each data packet subset includes at least one data packet, and each data packet subset corresponds to an image frame; The determining of N first queues respectively corresponding to the N data packet sets comprises: In response to detecting that the first queue includes a first storage address for a previous data packet, adding a first storage address for a current data packet to the first queue; wherein the current data packet and the previous data packet belong to the same data packet subset; and In response to detecting that the first queue includes a first storage address for a previous subset of data packets, adding a first storage address for a current subset of data packets to the first queue; wherein the current subset of data packets and the previous subset of data packets belong to the same data packet set.

5. The method according to claim 1, wherein: The data packet set includes at least one data packet subset, each data packet subset corresponds to an image frame; and sending the N first queues includes: In response to detecting that a target storage address in the first queue satisfies a predetermined condition, sending the target storage address; Wherein, the predetermined condition includes: the data packet indicated by the target storage address is used to determine an image frame.

6. The method according to claim 5, wherein: The predetermined condition also includes: the number of image frames corresponding to the N data packet sets is greater than or equal to a threshold.

7. The method according to claim 1, further comprising: In response to the plurality of target data packets in the first storage space being read, the plurality of target data packets are deleted from the first storage space.

8. The method according to claim 1, further comprising: Determine target adjustment parameters according to the scene identifier of the target scene; The target adjustment parameter includes at least one of the following: the target number of the image acquisition device and the target acquisition frequency of the image acquisition device; In response to detecting that the target adjustment parameter includes a target number of the image acquisition devices, adjusting the actual used number of the image acquisition devices according to the target number; as well as In response to detecting that the target adjustment parameter includes the target acquisition frequency of the image acquisition device, the actual acquisition frequency of the image acquisition device is adjusted according to the target acquisition frequency.

9. The method according to claim 1, wherein: The data packet includes a network layer abstraction unit.

10. An image processing method, comprising: Determine N second queues according to the N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of the data packets in the second storage space; Determining a plurality of data packets from the second storage space according to the second storage addresses in the N second queues; Determining a plurality of image frames according to the plurality of data packets; as well as The plurality of image frames are processed using a predetermined algorithm to obtain a processing result.

11. The method according to claim 10, wherein: The determining of N second queues according to the N first queues includes: Reading a plurality of target data packets from a first storage space according to the received first storage address in the first queue; storing the plurality of target data packets in a second storage area; Add the second storage addresses of the multiple target data packets in the second storage area to the second queue.

12. A video data stream processing device, comprising: A video data stream determination module, used to determine N video data streams from N image acquisition devices; wherein each video data stream includes at least one data packet, and N is an integer greater than or equal to 2; A division module, used for dividing the multiple data packets included in the N video data streams into N data packet sets, each data packet set corresponding to one video data stream; A first queue determination module, configured to determine N first queues corresponding to the N data packet sets respectively, each first queue comprising a first storage address in a first storage space of each data packet in the data packet set corresponding to the first queue; and A sending module is used to send the N first queues.

13. An image processing device, comprising: A second queue determination module, used to determine N second queues according to the N first queues; N is an integer greater than or equal to 2, and the N second queues include second storage addresses of the data packets in the second storage space; a data packet determining module, configured to determine a plurality of data packets from the second storage space according to the second storage addresses in the N second queues; An image frame determination module, used to determine a plurality of image frames according to the plurality of data packets; as well as The processing module is used to process the multiple image frames using a predetermined algorithm to obtain a processing result.

14. A video data stream processing device, comprising a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 9. 15 . An image processing device, comprising a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the method according to claim 10 .

16. An image processing system, comprising: N image acquisition devices, used for acquiring N video data streams; The video data stream processing device according to claim 14; as well as The image processing device as claimed in claim 15.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

18. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images