A video stream processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a video stream processing method and device, electronic equipment and storage medium. The method can include: transmitting a video stream at a current time to a preprocessor to cause the preprocessor to perform road element recognition on the video stream at the current time to obtain an element recognition result corresponding to the current time; sending the element recognition result corresponding to the current time to a main processor to cause the main processor to predict a video stream at a next time based on the element recognition result corresponding to the current time, determining first target allocation information in a case where a video stream prediction result represents that the predicted video stream at the next time is a first type of video stream, and awakening a coprocessor in a case where the coprocessor is not awakened. According to the technical scheme provided by the present disclosure, the video stream at the next time can be predicted, and in a case where it is determined that the video stream at the next time is the first type of video stream, the coprocessor is awakened in advance to improve the response timeliness of the coprocessor, i.e., the timeliness of processing data.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving, and in particular to a video stream processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rise of autonomous driving technology, intelligent driving and intelligent cockpit functions are gradually converging. Existing solutions include integrated cockpit controllers that combine intelligent cockpit and low-level driver assistance functions into a single controller. In current technologies, the main processor handles video streams with low computational loads, while the main processor and coprocessor jointly handle video streams with high computational loads. Existing solutions only call the coprocessor in scenarios requiring high computational loads. However, when the coprocessor is called, it needs to perform preparatory operations such as wake-up and computing power loading, which can easily lead to untimely response and consequently, untimely data processing. Summary of the Invention

[0003] This disclosure provides a video stream processing method, apparatus, electronic device, and storage medium to at least solve the problem in related technologies where a coprocessor is required but its response is untimely. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, a video stream processing method is provided, comprising:

[0005] The video stream at the current moment is transmitted to the preprocessor, so that the preprocessor can perform road element recognition on the video stream at the current moment and obtain the element recognition result corresponding to the current moment.

[0006] The element recognition result corresponding to the current moment is sent to the main processor so that the main processor can predict the video stream of the next moment based on the element recognition result corresponding to the current moment. If the video stream prediction result indicates that the predicted video stream of the next moment is a first type of video stream, the first target allocation information is determined, and if the coprocessor is not woken up, the coprocessor is woken up.

[0007] The first target allocation information indicates that, upon obtaining the actual video stream of the next moment, the actual video stream of the next moment will be allocated to the main processor and the coprocessor;

[0008] Receive the first target allocation information sent by the main processor.

[0009] In one possible implementation, the step of transmitting the video stream at the current moment to a preprocessor, so that the preprocessor performs road element recognition on the video stream at the current moment to obtain an element recognition result corresponding to the current moment, further includes:

[0010] The video stream at the current moment is transmitted to the preprocessor so that the preprocessor can perform road element recognition on the video stream at the current moment and obtain element recognition results corresponding to key elements; the key elements represent elements that cause the autonomous vehicle to avoid or stop.

[0011] In one possible implementation, the road elements include multiple elements that have an impact on autonomous vehicles;

[0012] The step of transmitting the current-time video stream to the preprocessor, so that the preprocessor performs road element recognition on the current-time video stream to obtain the element recognition result corresponding to the current time, further includes:

[0013] The video stream at the current moment is transmitted to the preprocessor so that the preprocessor performs frequency statistics on at least one element in the video stream at the current moment and obtains the frequency statistics result corresponding to the at least one element.

[0014] The element identification result is determined based on the frequency statistics.

[0015] In one possible implementation, sending the element recognition result corresponding to the current moment to the main processor, so that the main processor can predict the video stream for the next moment based on the element recognition result corresponding to the current moment, includes:

[0016] The element recognition result corresponding to the current moment is sent to the main processor, so that the main processor matches the element recognition result corresponding to the current moment with multiple preset results to obtain a target matching result; and predicts the video stream at the next moment based on the target matching result to obtain the video stream prediction result.

[0017] In one possible implementation, the method further includes:

[0018] The element recognition result corresponding to the current moment is sent to the main processor so that the main processor can predict the video stream of the next moment based on the element recognition result corresponding to the current moment. If the video stream prediction result indicates that the predicted video stream of the next moment is a second type of video stream, the second target allocation information is determined.

[0019] The second target allocation information indicates that, upon obtaining the actual video stream of the next moment, the actual video stream of the next moment will be allocated to the main processor.

[0020] In one possible implementation, the method further includes:

[0021] The video stream at the current moment is transmitted to the preprocessor so that the preprocessor performs blurring processing on the video stream at the current moment to obtain blurred video stream information; the blurred video stream information includes blurred video streams corresponding to multiple types of road elements.

[0022] The blurred video stream corresponding to each type of road element is assigned to the processor corresponding to each type of road element.

[0023] In one possible implementation, transmitting the current video stream to a preprocessor, such that the preprocessor performs blurring processing on the current video stream to obtain blurred video stream information, includes:

[0024] The video stream at the current moment is transmitted to the preprocessor; so that the preprocessor:

[0025] Extract multiple types of road elements and the corresponding image background information from the video stream at the current moment;

[0026] The background information of the images corresponding to the multiple types of road elements is blurred to obtain blurred video stream information corresponding to the multiple types of road elements.

[0027] According to a second aspect of the present disclosure, a video stream processing apparatus is provided, comprising:

[0028] The road element recognition module is used to transmit the video stream at the current moment to the preprocessor, so that the preprocessor can perform road element recognition on the video stream at the current moment and obtain the element recognition result corresponding to the current moment.

[0029] The first target allocation information determination module is used to send the element recognition result corresponding to the current moment to the main processor, so that the main processor can predict the video stream of the next moment based on the element recognition result corresponding to the current moment. If the video stream prediction result indicates that the predicted video stream of the next moment is a first type of video stream, the module determines the first target allocation information, and if the coprocessor is not woken up, the module wakes up the coprocessor.

[0030] A video stream allocation module is used to allocate the actual video stream of the next moment to the main processor and the coprocessor when the first target allocation information characterization is obtained;

[0031] The first target allocation information receiving module is used to receive the first target allocation information sent by the main processor.

[0032] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in any one of the first aspects above.

[0033] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided such that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described in the first aspect of the present disclosure.

[0034] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, cause a computer to perform the method described in any one of the first aspects of the present disclosure.

[0035] The technical solution provided by the embodiments of this disclosure brings at least the following beneficial effects: The video stream at the current moment is transmitted to a preprocessor, enabling the preprocessor to perform road element recognition on the video stream at the current moment, obtaining the element recognition result corresponding to the current moment; the element recognition result corresponding to the current moment is sent to the main processor, enabling the main processor to predict the video stream at the next moment based on the element recognition result corresponding to the current moment; if the video stream prediction result indicates that the predicted video stream at the next moment is a first type of video stream, first target allocation information is determined, and if the coprocessor is not woken up, the coprocessor is woken up; the first target allocation information indicates that, if the actual video stream at the next moment is obtained, the actual video stream at the next moment will be allocated to the main processor and the coprocessor; the first target allocation information sent by the main processor is received. According to the technical solution of this application, video stream type prediction is performed on the video stream at the current moment for the video stream at the next moment. If the video stream is characterized as a first type of video stream, the coprocessor is woken up in advance so that the coprocessor can respond in a timely manner to improve the timeliness of data processing.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0038] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment.

[0039] Figure 2This is a flowchart illustrating a video stream processing method according to an exemplary embodiment;

[0040] Figure 3 This is a flowchart illustrating an exemplary embodiment of determining element identification results based on frequency statistics.

[0041] Figure 4 This is a flowchart illustrating a method for obtaining video stream prediction results according to an exemplary embodiment;

[0042] Figure 5 This is a flowchart illustrating, according to an exemplary embodiment, the allocation of a blurred video stream to a corresponding processor;

[0043] Figure 6 This is a flowchart illustrating an exemplary embodiment for obtaining blurred video stream information;

[0044] Figure 7 This is a schematic diagram of a video stream processing system according to an exemplary embodiment;

[0045] Figure 8 This is a block diagram of a video stream processing apparatus according to an exemplary embodiment;

[0046] Figure 9 This is a schematic diagram of an electronic device for video streaming processing according to an exemplary embodiment. Detailed Implementation

[0047] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0048] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0049] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include: a main processor, a preprocessor, a preprocessor, and a coprocessor.

[0050] In an optional embodiment, in an autonomous driving scenario, the vehicle transmits road video streams captured by its onboard camera to a front-end processor. The front-end processor forwards the video stream to a preprocessor for preprocessing. The front-end processor then forwards the preprocessor's calculation results to the main processor for decision-making. The main processor compares the received results with preset results and generates corresponding allocation information based on the comparison results, which is then transmitted to the front-end processor. The front-end processor allocates resources to the main processor and coprocessors according to the allocation information.

[0051] It should be noted that the following diagram shows one possible sequence of steps, and it is not strictly necessary to follow this order. Some steps can be executed in parallel without interdependence.

[0052] Before introducing the method embodiments provided in this application, a brief introduction will be given on the application scenarios, related terms or nouns that may be involved in the method embodiments of this application, so as to facilitate the understanding of those skilled in the art.

[0053] Figure 2 This is a flowchart illustrating a video stream processing method according to an exemplary embodiment. Figure 2 As shown, the process may include the following steps. The executing entity may be a module with forwarding identification capabilities, such as the aforementioned preprocessor.

[0054] S201, the video stream at the current moment is transmitted to the preprocessor so that the preprocessor can perform road element recognition on the video stream at the current moment and obtain the element recognition result corresponding to the current moment.

[0055] In the embodiments of this specification, the front-end processor may be a field-programmable gate array (FPGA); the preprocessor may refer to a processor capable of processing various types of data collected by the vehicle. The preprocessor may be a processor independent of the main processor, specifically an embedded neural network processor (NPU), or it may be integrated with the main processor, representing a preprocessing module inside the main processor, specifically a central processing unit (CPU) or an NPU module inside the main processor; road elements may refer to elements that can constitute the road environment, such as traffic lights, pedestrians, vehicles, obstacles, and road signs.

[0056] In one possible implementation, a preprocessor can transmit the current-time video stream captured by the front-facing camera, left and right cameras, rear-view camera, and cabin monitoring camera of the autonomous vehicle to a preprocessor, so that the preprocessor can perform road element recognition on the current-time video stream and obtain the element recognition result corresponding to the current time.

[0057] In another possible implementation, the video stream at the current moment first passes through a deserializer, which is an interface circuit in high-speed data communication, independent of the main processor, preprocessor, front-end processor, and coprocessor. The deserializer converts the video stream at the current moment into a parallel video data stream such as the Mobile Industry Processor Interface (MIPI) and transmits it to the preprocessor, so that the preprocessor can perform road element recognition on the video stream at the current moment and obtain the element recognition result corresponding to the current moment.

[0058] The preprocessor identifies road elements in the video stream at the current moment, classifies different types of road elements, and performs frequency statistics on the same road element while classifying the road element categories, thus obtaining the identification results of the element type and the identification results of the element frequency statistics corresponding to the current moment.

[0059] By using a preprocessor to identify road elements and obtain information about various elements on the road, we can analyze these elements and gain a better understanding of the road conditions.

[0060] S203, the element identification result corresponding to the current moment is sent to the main processor so that the main processor can predict the video stream of the next moment based on the element identification result corresponding to the current moment. If the video stream prediction result indicates that the predicted video stream of the next moment is a first type of video stream, the first target allocation information is determined, and if the coprocessor is not woken up, the coprocessor is woken up.

[0061] In the embodiments of this specification, the main processor can refer to a processor capable of performing central decision-making operations, specifically, it can be a System-on-a-Chip (SOC); the first type of video stream can refer to a video stream in which the frequency of road element occurrence is greater than or equal to a set threshold, wherein the set threshold can be H, and H can be 10, 20 or 30, which is not limited in this application; the first target allocation information can refer to the specific resolution and frame rate data information allocated to the main processor and the coprocessor, wherein the sum of the number of frames processed per second by the main processor and the number of frames processed per second by the coprocessor equals the frame rate, specifically, for example, if the image resolution and the image frame rate are 108030p, that is, the image resolution is 1080 and the image frame rate is 30 frames / s, the first target allocation information can be to allocate 10 frames / s to the main processor and 20 frames / s to the coprocessor, and the resolution can remain unchanged, which is not limited in this application; the next moment can refer to the next moment that exists for a certain period of time from the current moment, and does not necessarily represent the next moment immediately following the current moment; the coprocessor can include one or more coprocessors independent of the main processor, specifically, it can be an NPU.

[0062] In one possible implementation, the element identification result corresponding to the current moment is sent to the main processor, so that the main processor compares the element identification result corresponding to the current moment in the database, predicts the video stream of the next moment based on the database comparison result, determines the first target allocation information if the video stream prediction result indicates that the predicted video stream of the next moment is a first type of video stream, and wakes up the coprocessor if the coprocessor is not woken up.

[0063] In another possible implementation, the element identification result corresponding to the current moment is sent to the main processor so that the main processor can predict the video stream of the next moment based on the element identification result corresponding to the current moment. If the video stream prediction result indicates that the predicted video stream of the next moment is a first type of video stream, a wake-up signal is sent to the coprocessor to wake up the coprocessor.

[0064] The calculation results are sent to the main processor for decision-making. When the main processor detects a video stream that is identified as the first type in the pre-set characteristics, it issues the corresponding allocation information and wakes up the coprocessor in advance. This allows the coprocessor to prepare in advance before the first type of video stream actually arrives, avoiding untimely response from the coprocessor.

[0065] S205, the first target allocation information indicates that, upon obtaining the actual video stream of the next moment, the actual video stream of the next moment will be allocated to the main processor and the coprocessor.

[0066] In the embodiments described in this specification, the first target allocation information may refer to the specific resolution and frame rate data information allocated to the main processor and the coprocessor.

[0067] Specifically, the first target allocation information can be the ratio of the frame rate allocated to the main processor and the coprocessor according to the key element. For example, when the key element is a red light, the video stream frame rate can be allocated to the main processor and the coprocessor in a ratio of 3:7. When the key element is a pedestrian, the video stream frame rate can be allocated to the main processor and the coprocessor in a ratio of 2:3.

[0068] In one possible implementation, the actual video stream of the next moment is allocated to the main processor and the coprocessor. After the coprocessor performs collaborative calculations, the calculation results are returned to the main processor for final decision-making.

[0069] If the video stream prediction result indicates that the next moment belongs to the first type of video stream, when the actual video stream of the next moment is obtained, it is allocated to the main processor and coprocessor according to the instructions of the main processor, so that the coprocessor can start working in advance to process and calculate the video stream. After the coprocessor performs advance calculations on the video stream, it can optionally call up the algorithm that matches the relevant elements in the video stream in advance, thereby reducing the response time of importing future high-bandwidth video data streams.

[0070] S207, receive the first target allocation information sent by the main processor.

[0071] In one possible implementation, the first target allocation information sent by the main processor can be received by the front-end processor.

[0072] It should be noted that after receiving the first target allocation information sent by the main processor and allocating the actual video stream of the next moment to the coprocessor, the coprocessor performs the calculations, and the calculation result is returned to the main processor for central decision-making.

[0073] The preprocessor has the ability to identify and forward video streams, and distributes the video streams to the main processor and coprocessors according to the instructions issued by the main processor, so that the coprocessors can start working in advance.

[0074] The video stream at the current moment is transmitted to the preprocessor, which performs road element recognition on the video stream at the current moment to obtain the element recognition result corresponding to the current moment. This also includes:

[0075] The video stream at the current moment is transmitted to the preprocessor so that the preprocessor can perform road element recognition on the video stream at the current moment and obtain the element recognition result corresponding to the key element; the key element represents the element that causes the autonomous vehicle to avoid or stop.

[0076] In the embodiments of this specification, key elements represent elements that cause autonomous vehicles to avoid or stop, such as traffic lights, pedestrians, road obstacles, etc.

[0077] In one possible implementation, the video stream at the current moment is transmitted to the preprocessor so that the preprocessor can perform road element identification on the video stream at the current moment. The preprocessor performs road element identification on the video stream at the current moment according to different types of pre-set key elements, searches for whether there are road elements that are the same as the different types of pre-set key elements, and obtains the element identification result corresponding to the key elements.

[0078] For example, pre-set key elements may include red lights, pedestrians, trees, traffic cones, oil tankers, etc. The preprocessor performs road element recognition on the video stream at the current moment. If a red light or vehicle is found in the video stream at the current moment, it represents the element recognition result corresponding to the key element, i.e., red light.

[0079] The preprocessor identifies key elements that can affect vehicle movement and increase the computational load of the video stream in the next moment. The preprocessor's identification of these elements serves as a reminder, prompting the main processor to provide timely response strategies and improve the accuracy of predicting the video stream in the next moment.

[0080] Furthermore, the non-critical elements in the embodiments of this specification may also include vehicles, road signs, and lane lines, etc. These non-critical elements can be used as some processing basis in autonomous driving scenarios, but they are not included in the critical elements.

[0081] like Figure 3 As shown, Figure 3 This is a flowchart illustrating an element identification result based on frequency statistics, according to an exemplary embodiment. In another possible implementation, the above-described S201 may include the following steps:

[0082] S301, the video stream at the current moment is transmitted to the preprocessor so that the preprocessor performs frequency statistics on at least one element in the video stream at the current moment and obtains the frequency statistics result corresponding to the at least one element.

[0083] In one possible implementation, the video stream at the current moment is transmitted to the preprocessor so that the preprocessor performs frequency statistics on at least one element in the video stream at the current moment. The preprocessor counts the frequency of each element in the video stream at the current moment and obtains the frequency statistics result corresponding to at least one element.

[0084] For example, the preprocessor counts the frequency of different types of elements in the video stream at the current moment. For instance, if it counts that a vehicle appears ten times in the video stream at the current moment, it obtains the frequency statistics corresponding to the vehicle, i.e., the road element vehicle appears ten times per frame.

[0085] S303, determine the element identification result based on the frequency statistics result.

[0086] In one possible implementation, the element identification result is determined based on frequency statistics.

[0087] The system identifies situations such as heavy traffic on the road. If there are more vehicles on the road, the computational load of the video stream in future moments will increase, and the preprocessor will identify road elements more frequently. As a reminder, the preprocessor will provide the main processor with timely response strategies to improve the accuracy of video stream prediction for the next moment.

[0088] like Figure 4 As shown, Figure 4 This is a flowchart illustrating a method for obtaining a video stream prediction result according to an exemplary embodiment. In another possible implementation, the above-described S203 may include the following steps:

[0089] S401, the element recognition result corresponding to the current time is sent to the main processor, so that the main processor can perform result matching with multiple preset results based on the element recognition result corresponding to the current time to obtain the target matching result.

[0090] In the embodiments described in this specification, the preset result may refer to various different road element conditions and combinations of different road elements pre-stored in the main processor. This may include the presence of road element red lights, road element vehicle frequency exceeding a threshold, road element roadblocks, road element pedestrians, road element road signs, and different combinations thereof. This application does not limit this. The target matching result may be a match or a non-match.

[0091] Specifically, as shown in Table 1, the element recognition result corresponding to the current time is sent to the main processor so that the main processor can match the element recognition result corresponding to the current time with multiple preset results to obtain the target matching result.

[0092] S403, based on the target matching result, predict the video stream at the next moment to obtain the video stream prediction result.

[0093] In the embodiments of this specification, the video stream prediction result may refer to a first type of video stream and a second type of video stream.

[0094] Specifically, as shown in Tables 1 and 2, the video stream at the next moment is predicted based on the target matching results to obtain the video stream prediction results.

[0095] Table 1

[0096] Element recognition results Preset results Target matching results Video stream prediction results red light red light match Type 1 video stream Red light + pedestrians Red light + pedestrians match Type 1 video stream Pedestrians + Vehicles >= Threshold Pedestrians + Vehicles >= Threshold match Type 1 video stream pedestrian pedestrian match Type 1 video stream obstacle obstacle match Type 1 video stream ... ... ... ...

[0097] The threshold can be 10, 20 or 25, and this application does not limit it.

[0098] Table 2

[0099]

[0100]

[0101] The threshold can be 2 or 3, and this application does not limit it.

[0102] If the element recognition result contains a key element, the video stream prediction result is a first-type video stream. If the element recognition result contains a non-key element, the video stream prediction result is a second-type video stream. For example, if the element recognition result is a red light, the video stream prediction result is a first-type video stream. If the element recognition result is lane lines and road signs, the video stream prediction result is a second-type video stream.

[0103] The main processor provides corresponding video stream prediction results based on the element recognition results sent by the preprocessor. The main processor itself has preset results, which are compared with the element recognition results sent by the preprocessor to determine the video stream prediction results. Adaptive predictions are made for different situations.

[0104] The video stream processing method further includes: sending the element recognition result corresponding to the current moment to the main processor, so that the main processor can predict the video stream of the next moment based on the element recognition result corresponding to the current moment, and determining the second target allocation information when the video stream prediction result indicates that the predicted video stream of the next moment is a second type of video stream.

[0105] In the embodiments described in this specification, the second target allocation information may refer to the specific resolution and frame rate data information allocated to the main processor.

[0106] Specifically, the second target allocation information can be data information that determines the frame rate allocated to the main processor based on road elements. For example, when the road elements are lane lines and road signs, the entire frame rate of the video stream can be allocated to the main processor.

[0107] In the embodiments of this specification, the second type of video stream may refer to a video stream in which road elements appear at a frequency less than a set threshold.

[0108] In one possible implementation, the element identification result corresponding to the current moment is sent to the main processor, so that the main processor compares the element identification result corresponding to the current moment in the database, predicts the video stream of the next moment based on the database comparison result, and determines the second target allocation information if the video stream prediction result indicates that the predicted video stream of the next moment is a second type of video stream.

[0109] The calculation results are sent to the main processor for decision-making. When the main processor detects a video stream that is identified as the second type in the pre-set characteristics, it issues the corresponding allocation information. The main processor can be used for calculation when its computing power meets the requirements.

[0110] like Figure 5 As shown, Figure 5 This is a flowchart illustrating, according to an exemplary embodiment, the allocation of a blurred video stream to a corresponding processor. In another possible implementation, the above-described transmission of the current-moment video stream to the preprocessor may include the following steps:

[0111] S501, the video stream at the current moment is transmitted to the preprocessor; so that the preprocessor performs blurring processing on the video stream at the current moment to obtain blurred video stream information; the blurred video stream information includes blurred video streams corresponding to multiple types of road elements.

[0112] In the embodiments of this specification, blurring can refer to blurring the background image information of foreground elements, which is a processing method that can compress video streams; blurred video streams can refer to video streams obtained after blurring the background of foreground elements in the original video stream.

[0113] Blurring can be achieved by applying a low frame rate, low resolution, and H.264 / 265 compressed bitstream to the actual video stream at the next moment.

[0114] In one possible implementation, the preprocessor identifies road elements in the video stream at the current moment, separates different types of road elements, i.e. foreground elements, from the background image, and blurs the background image information to obtain blurred video stream information.

[0115] S503, the blurred video stream corresponding to each type of road element is assigned to the processor corresponding to each type of road element.

[0116] In one possible implementation, the blurred video stream corresponding to each type of road element is assigned to the processor corresponding to each type of road element.

[0117] For example, a blurred video stream of road element vehicles is assigned to a processor that processes road elements as vehicles, wherein the processor that processes road elements as vehicles has an algorithm for processing vehicles; and a blurred video stream of road element pedestrians is assigned to a processor that processes road elements as pedestrians, wherein the processor that processes road elements as pedestrians has an algorithm for processing pedestrians.

[0118] The preprocessor blurs the video stream, and the pre-processor distributes the blurred results of different types of elements to the corresponding processors. Each processor has a corresponding algorithm for different road elements. After extraction and allocation, each processor can be responsible for processing the road elements under its algorithm, making the operation faster and more efficient. Secondly, targeted blurring of the video stream can reduce the amount of computation.

[0119] like Figure 6 As shown, Figure 6 This is a flowchart illustrating an exemplary embodiment for obtaining blurred video stream information. In another possible implementation, the video stream at the current moment is transmitted to a preprocessor so that the preprocessor performs blurring processing on the video stream at the current moment to obtain blurred video stream information, which may include the following steps:

[0120] S601, the video stream at the current moment is transmitted to the preprocessor; so that the preprocessor: extracts multiple types of road elements and image background information corresponding to the multiple types of road elements in the video stream at the current moment.

[0121] In the embodiments of this specification, the image background information for any element can be all the information in the original image except for that element.

[0122] In one possible implementation, the preprocessor extracts multiple types of road elements and corresponding image background information from the video stream at the current moment.

[0123] For example, the preprocessor extracts two road elements, M1 and M2, that appear simultaneously or alternately in an image frame, as well as the corresponding image background information N1 and N2. The image background information N1 of M1 can be all information except road element M1, including road element M2; the image background information N2 of M2 can be all information except road element M2, including road element M1.

[0124] S603, blur the image background information corresponding to the multiple types of road elements respectively to obtain blurred video stream information corresponding to the multiple types of road elements.

[0125] In one possible implementation, the image background information corresponding to multiple types of road elements is blurred to obtain blurred video stream information corresponding to multiple types of road elements.

[0126] For example, for image background information N1, road element M2, as part of the image background information corresponding to road element M1, is blurred to generate a new image block M'2, which is then filled back into image background information N1 to generate new image background information N'1 for road element M1; similarly, for image background information N2, road element M1, as part of the image background information corresponding to road element M2, is blurred to generate a new image block M'1, which is then filled back into image background information N2 to generate new image background information N'2 for road element M2, thus obtaining blurred video stream information corresponding to the two types of road elements.

[0127] The preprocessor identifies each type of element and its image background information, performs blurring processing to eliminate the influence of irrelevant image elements, and blurs the image background information to reduce the amount of computation.

[0128] Figure 7 This is a schematic diagram of a video stream processing system according to an exemplary embodiment. (Refer to...) Figure 7 The system may include:

[0129] The system includes a deserializer, a main processor integrated SoC chip, a first coprocessor NPU, a second coprocessor NPU, a front-end processor FPGA, an Ethernet conversion chip, a functional safety unit MCU, an ETH bus, and a CAN bus.

[0130] The deserializer is the interface circuit in high-speed data communication; the main processor integrated SoC chip generally has CPU, GPU, and NPU processing units, and can run Android and Linux operating systems simultaneously to realize smart cockpit and low-level assisted driving functions; the first coprocessor NPU and the second coprocessor NPU are used to process network application data packets, and adopt a data-driven parallel computing architecture, which can be used to process massive multimedia data such as video and images; the front-end processor FPGA has identification and forwarding functions; the Ethernet conversion chip ETH Switch is generally used to forward Ethernet data packets through the Ethernet data bus; the functional safety unit MCU is used to interface with the vehicle control unit outside the computing platform, sending the calculation results of the integrated SoC to the vehicle control unit for vehicle control, and also carrying some planning and control algorithms for vehicle control decision-making, etc.; the CAN bus, through Ethernet cascading, uses the functional MCU to issue vehicle control execution commands and interfaces with the vehicle control unit through the CAN bus.

[0131] MIPI (Mobile Industry Processor Interface) signals are typically converted from the serial raw video signals captured by the camera into parallel video data signals by a deserializer, which are then used by subsequent FPGAs, SoCs, NPUs, etc. for further video analysis and processing.

[0132] The main processor, such as a converged SoC, communicates with the coprocessor, such as the NPU, via an Ethernet data bus or a PCIe (Peripheral Component Interconnect Express) data bus. For example, the main processor may distribute video data that it cannot process in a timely manner to the coprocessor via the PCIe bus or Ethernet data bus for AI calculations. The calculation results are then returned to the main processor via the aforementioned bus for further central decision-making.

[0133] The aforementioned Ethernet data bus typically uses an Ethernet switch chip to forward Ethernet data packets.

[0134] Multiple Ethernet buses can be selected from multiple RGMII (Reduced Gigabit Media Independent Interface), which is a type of on-board Ethernet data bus, or multiple SGMII (Serial Gigabit Media Independent Interface). The aforementioned Ethernet switch chip acts as a data forwarding hub, connecting the main processor, coprocessor, functional safety unit (MCU), and front-end processor (FPGA), etc.

[0135] The raw video stream can be sent directly to the main processor and coprocessor via MIPI signal, or it can be preprocessed by the front-end processor FPGA before being sent to the main processor and coprocessor.

[0136] Figure 8 This is a block diagram of a video stream processing apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The device may include:

[0137] The road element recognition module 801 is used to transmit the video stream at the current moment to the preprocessor so that the preprocessor can perform road element recognition on the video stream at the current moment and obtain the element recognition result corresponding to the current moment.

[0138] The first target allocation information determination module 803 is used to send the element recognition result corresponding to the current moment to the main processor so that the main processor can predict the video stream of the next moment based on the element recognition result corresponding to the current moment. When the video stream prediction result indicates that the predicted video stream of the next moment is a first type of video stream, the first target allocation information is determined, and the coprocessor is woken up when the coprocessor is not woken up.

[0139] The video stream allocation module 805 is used to allocate the actual video stream of the next moment to the main processor and the coprocessor when the first target allocation information characterization is obtained.

[0140] The first target allocation information receiving module 807 is used to receive the first target allocation information sent by the main processor.

[0141] In one possible implementation, the road element recognition module 801 may further include:

[0142] The key element identification unit is used to transmit the current video stream to the preprocessor so that the preprocessor can identify road elements in the current video stream and obtain the element identification results corresponding to the key elements; the key elements represent the elements that cause the autonomous vehicle to avoid or stop.

[0143] In one possible implementation, the road element recognition module 801 may further include:

[0144] The frequency statistics result acquisition unit is used to transmit the video stream at the current moment to the preprocessor so that the preprocessor can perform frequency statistics on at least one element in the video stream at the current moment and obtain the frequency statistics result corresponding to at least one element.

[0145] An element identification result determination unit is used to determine the element identification result based on frequency statistics results.

[0146] In one possible implementation, the first target allocation information determination module 803 may further include:

[0147] The target matching result acquisition unit is used to send the element recognition result corresponding to the current time to the main processor, so that the main processor can match the result with multiple preset results based on the element recognition result corresponding to the current time to obtain the target matching result.

[0148] The video stream prediction result acquisition unit is used to predict the video stream at the next moment based on the target matching result, and obtain the video stream prediction result.

[0149] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0150] Figure 9 This is a block diagram illustrating an electronic device for video stream processing according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video streaming method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0151] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a video streaming processing method as described in the embodiments of this disclosure.

[0153] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform a video streaming processing method according to an embodiment of the present disclosure. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0154] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform a method for video stream processing according to an embodiment of the present disclosure.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0156] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0157] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for processing a video stream, characterized by, The method comprises: transmitting a video stream of a current time to a preprocessor, so that the preprocessor performs road element recognition on the video stream of the current time to obtain an element recognition result corresponding to the current time; sending the element recognition result corresponding to the current time to a main processor, so that the main processor predicts a video stream of a next time based on the element recognition result corresponding to the current time, determines first target allocation information in a case where a video stream prediction result represents that the predicted video stream of the next time is a first type of video stream, and wakes up a coprocessor in a case where the coprocessor is not woken up; wherein the video stream prediction result is the first type of video stream if the element recognition result contains a key element; the key element represents an element that causes an autonomous vehicle to avoid or stop; the first target allocation information represents that, in a case where an actual video stream of the next time is obtained, the actual video stream of the next time is allocated to the main processor and the coprocessor; the first target allocation information is proportional data information of a frame rate of the main processor and the coprocessor allocated according to the key element; receiving the first target allocation information sent by the main processor.

2. The video stream processing method of claim 1, wherein, The road elements include a plurality of elements that have an impact on the autonomous vehicle; The method further comprises: transmitting the video stream of the current time to the preprocessor, so that the preprocessor performs frequency statistics on at least one element in the video stream of the current time to obtain a frequency statistical result corresponding to the at least one element; determining the element recognition result based on the frequency statistical result.

3. The video stream processing method of claim 1, wherein, The method further comprises: sending the element recognition result corresponding to the current time to the main processor, so that the main processor performs result matching based on the element recognition result corresponding to the current time and a plurality of preset results to obtain a target matching result; predicting the video stream of the next time based on the target matching result to obtain the video stream prediction result.

4. The video stream processing method of claim 1, wherein, The method further comprises: sending the element recognition result corresponding to the current time to the main processor, so that the main processor predicts the video stream of the next time based on the element recognition result corresponding to the current time, and determines second target allocation information in a case where a video stream prediction result represents that the predicted video stream of the next time is a second type of video stream; the second target allocation information represents that, in a case where an actual video stream of the next time is obtained, the actual video stream of the next time is allocated to the main processor.

5. The method of claim 1, wherein, The method further comprises: transmit the current time video stream to a preprocessor; so that the preprocessor blurs the current time video stream to obtain blurred video stream information; the blurred video stream information includes a plurality of types of road elements each corresponding to a blurred video stream; Assign each type of road element corresponding to the blurred video stream to the processor corresponding to each type of road element.

6. The video stream processing method of claim 5, wherein, The current time video stream is transmitted to the preprocessor; So that the preprocessor blurs the current time video stream to obtain blurred video stream information includes: Transmit the current time video stream to a preprocessor; so that the preprocessor: Extract a plurality of types of road elements in the current time video stream and image background information corresponding to the plurality of types of road elements; Blurred video stream information corresponding to the plurality of types of road elements is obtained by blurring the image background information corresponding to the plurality of types of road elements respectively.

7. A video stream processing apparatus, characterized by comprising: Comprising: The road element recognition module is used for transmitting the current time video stream to the preprocessor, so that the preprocessor performs road element recognition on the current time video stream to obtain the element recognition result corresponding to the current time; The first target allocation information determination module is used for sending the element recognition result corresponding to the current time to the main processor, so that the main processor predicts the next time video stream based on the element recognition result corresponding to the current time, determines the first target allocation information in the case that the video stream prediction result represents that the predicted video stream of the next time is a first type of video stream, and wakes up the coprocessor in the case that the coprocessor is not awakened; wherein, if the element recognition result contains a key element, the video stream prediction result is a first type of video stream; the key element represents an element that makes the autonomous vehicle avoid or stop; The video stream allocation module is used for allocating the actual video stream of the next time to the main processor and the coprocessor in the case that the first target allocation information represents that the actual video stream of the next time is obtained; the first target allocation information is proportional data information of frame rate allocated to the main processor and the coprocessor according to the key element; The first target allocation information receiving module is used for receiving the first target allocation information sent by the main processor.

8. An electronic device, comprising: Comprising: A processor; A memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video stream processing method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the video stream processing method of any one of claims 1 to 6. When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the video stream processing method of any one of claims 1 to 6.

Images