Automatic driving computing power early warning method and device, computer readable storage medium

Abstract

Embodiments of the present application provide an automatic driving computing power early warning method and device, and a computer readable storage medium, wherein the method comprises: collecting first video information during driving; decoding the first video information to obtain a first video frame; wherein the first video frame is an image for recognition analysis; analyzing and recognizing the first video frame based on a preset algorithm model to determine the type of the first video frame; if the first video frame is a warning frame, uploading the first video frame and / or subsequent video frames of the first video frame to a server for operation processing to obtain a calculation result; and adjusting the speed and direction during driving based on the calculation result to perform automatic driving. In the above scheme, an algorithm calling mechanism with the server is established in advance, ensuring that the vehicle can continuously maintain the algorithm unit meeting the analysis requirements during high-speed driving, while saving traffic bandwidth.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to an autonomous driving computing power early warning method and device, and a computer-readable storage medium. Background Technology

[0002] With the rise of autonomous driving technology, onboard computing units have gradually become key components in automotive driver assistance systems. These units primarily process data collected by sensors such as cameras and millimeter-wave radar, assisting in the analysis and judgment of factors including distance and speed of vehicles ahead and behind, road signs, traffic lights, lanes, and obstacles. This provides crucial information for intelligent decision-making in autonomous and semi-autonomous driving systems.

[0003] Existing computing units are generally built into vehicles, or connected to cloud platform computing servers via 5G networks, leveraging the server's powerful computing capabilities for faster and more complex image and data processing. The advantage of using onboard computing units is that data is processed locally, resulting in good real-time performance. The disadvantage is that the computing power of onboard units is limited, making it difficult to meet sudden increases in computing demands. Cloud-based computing servers generally have high computational and analytical capabilities, but for moving vehicles, data interaction must be achieved via wireless communication protocols such as 5G, consuming significant bandwidth resources. Summary of the Invention

[0004] This invention provides an autonomous driving computing power early warning method and device, and a computer-readable storage medium, which reduces bandwidth consumption and improves computing speed.

[0005] The technical solution of this invention is implemented as follows:

[0006] This invention provides a method for early warning of computing power in autonomous driving, the method comprising:

[0007] Collect first video information during driving;

[0008] Based on the first video information, decoding processing is performed to obtain the first video frame; wherein, the first video frame is an image used for recognition and analysis;

[0009] Based on the first video frame, the type of the first video frame is determined by analysis and identification using a preset algorithm model; wherein, the preset algorithm model is a model for image judgment and recognition; the type of the first video frame includes a calculation frame and a warning frame, wherein the calculation frame is an image during normal driving, and the warning frame is an image indicating an impending surge in computing power;

[0010] If the first video frame is a warning frame, then the first video frame and / or subsequent video frames of the first video frame are uploaded to the server for processing to obtain the calculation result.

[0011] Based on the calculation results, the speed and direction during driving are adjusted to achieve automatic driving.

[0012] In the above scheme, the step of analyzing and identifying the type of the first video frame based on the first video frame using a preset algorithm model includes:

[0013] The first video frame is input into the preset algorithm model for analysis and recognition processing to obtain the labeling result of the first video frame; wherein, the labeling result includes a label value;

[0014] If the marker value is greater than the first preset threshold, then the first video frame is determined to be a warning frame;

[0015] If the marker value is not greater than the first preset threshold, then the first video frame is determined to be a calculation frame.

[0016] In the above scheme, if the first video frame is a warning frame, then the first video frame is uploaded to the server for calculation and processing to obtain the calculation result, including:

[0017] If the first video frame is a warning frame, a local connection is established with the server via the network;

[0018] The first video frame and / or subsequent video frames of the first video frame are transmitted to the server through the network for processing to obtain the calculation result from the server.

[0019] In the above scheme, before determining the type of the first video frame by analyzing and identifying it using a preset algorithm model based on the first video frame, the method further includes:

[0020] Acquire training sample data; the sample data is a first warning frame; wherein, the first warning frame includes the image corresponding to the forward time point under the condition of a sudden increase in computing power;

[0021] Based on the first warning frame, the initial algorithm model is continuously trained to determine the preset algorithm model.

[0022] In the above scheme, obtaining the training sample data includes:

[0023] Acquire second video information from the historical driving process and the central processing unit's computational allocation records;

[0024] Based on the computation allocation record, a filtering process is performed to obtain the first computation allocation record where computation is allocated to two computing power units;

[0025] Obtain the third video information corresponding to the first operation allocation record from the second video information;

[0026] The third video information is decoded to obtain a third video frame, which is then used as the sample data.

[0027] In the above scheme, the first warning frame is divided into a training warning frame and a test warning frame;

[0028] The step of continuously training the initial algorithm model based on the first warning frame to determine the preset algorithm model includes:

[0029] Based on the training warning frames, the initial algorithm model is trained to obtain a training model;

[0030] Based on the test warning frame, the training model is tested and optimized to determine the preset algorithm model.

[0031] In the above scheme, the step of optimizing the training model based on the test warning frame to determine the preset algorithm model includes:

[0032] The test warning frame is input into the training model to obtain the training labeling result of the test warning frame; the training labeling result includes training labeling values;

[0033] If the training label value is greater than the second preset threshold, then the preset algorithm model is obtained;

[0034] If the training label value is not greater than the second preset threshold, then training continues until the training label value in the training label result of the training model is greater than the first preset threshold, and then the preset algorithm model is obtained.

[0035] The method in the above scheme further includes:

[0036] Obtain the preset algorithm model from the server.

[0037] In the above scheme, after determining the type of the first video frame by analyzing and identifying it using a preset algorithm model based on the first video frame, the method further includes:

[0038] If the first video frame is a calculation frame, then driving calculation processing is performed locally to obtain the calculation processing load;

[0039] If the computational load exceeds a third preset threshold, the computational task is assigned to a local backup computing unit for computation to obtain the computational result.

[0040] This invention provides an autonomous driving computing power early warning device, which includes: a data acquisition unit, an acquisition unit, a determination unit, and a judgment unit; wherein,

[0041] The acquisition unit is used to acquire first video information during driving;

[0042] The acquisition unit is used to perform decoding processing based on the first video information to obtain a first video frame; wherein, the first video frame is an image used for recognition and analysis;

[0043] The determining unit is used to analyze and identify the type of the first video frame based on the first video frame using a preset algorithm model; wherein the preset algorithm model is a model for image judgment and recognition; the type of the first video frame includes a calculation frame and a warning frame, wherein the calculation frame is an image during normal driving, and the warning frame is an image indicating an impending surge in computing power;

[0044] The judgment unit is configured to, if the first video frame is a warning frame, upload the first video frame and / or subsequent video frames of the first video frame to the server for calculation and processing to obtain a calculation result; and adjust the speed and direction during the driving process according to the calculation result to perform automatic driving.

[0045] This invention provides an autonomous driving computing power early warning device, the autonomous driving computing power early warning device comprising:

[0046] Memory, used to store executable instructions;

[0047] A processor is configured to execute executable instructions stored in the memory. When the executable instructions are executed, the processor executes the autonomous driving computing power warning method.

[0048] This invention provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the autonomous driving computing power warning method.

[0049] This invention provides an autonomous driving computing power early warning method and device, and a computer-readable storage medium. The method includes: acquiring first video information during driving; decoding the first video information to obtain a first video frame; wherein the first video frame is an image used for recognition and analysis; analyzing and identifying the first video frame using a preset algorithm model to determine the type of the first video frame; wherein the preset algorithm model is a model for image judgment and recognition; the type of the first video frame includes a calculation frame and an early warning frame, wherein the calculation frame is an image during normal driving, and the early warning frame is an image indicating an impending surge in computing power; if the first video frame is an early warning frame, uploading the first video frame and / or subsequent video frames of the first video frame to a server for computation and processing to obtain a calculation result; and adjusting the speed and direction during driving based on the calculation result to perform autonomous driving. In this scheme, a computing power call mechanism with the server is pre-established, ensuring that the vehicle can continuously maintain computing power units sufficient to meet analysis needs during high-speed driving, improving the computing speed while saving bandwidth. Attached Figure Description

[0050] Figure 1 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 1 ;

[0051] Figure 2 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 2 ;

[0052] Figure 3 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 3 ;

[0053] Figure 4 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 4 ;

[0054] Figure 5 This invention provides an optional scheme illustration of an autonomous driving computing power early warning method. Figure 1 ;

[0055] Figure 6 This invention provides an optional scheme illustration of an autonomous driving computing power early warning method. Figure 2 ;

[0056] Figure 7 A schematic diagram of the structure of an autonomous driving computing power early warning device provided in an embodiment of the present invention. Figure 1 ;

[0057] Figure 8 A schematic diagram of the structure of an autonomous driving computing power early warning device provided in an embodiment of the present invention. Figure 2 . Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0059] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 1 , will combine Figure 1 The steps shown are explained.

[0060] S101, Collect the first video information during driving.

[0061] In some embodiments of the present invention, the present invention is applicable to autonomous driving scenarios.

[0062] In some embodiments of the present invention, the terminal can acquire first video information of the external environment during autonomous driving by activating the acquisition equipment.

[0063] It should be noted that the first video information is acquired in real time; and the first video information includes real-time information on vehicles, pedestrians, signs, and roads on the road. The acquisition device can be a camera, but the present invention is not limited thereto.

[0064] S102. Based on the first video information, perform decoding processing to obtain the first video frame.

[0065] In some embodiments of the present invention, the decoding process is to convert continuous video into frames of images, and the first video frame is an image used for recognition and analysis.

[0066] In some embodiments of the present invention, the terminal can perform decoding processing on the first video information, convert the video into an image, and obtain the first video frame corresponding to the first video information.

[0067] S103. Based on the first video frame, analyze and identify it using a preset algorithm model to determine the type of the first video frame.

[0068] In some embodiments of the present invention, the preset algorithm model is a model for image judgment and recognition, which is a pre-trained model. The preset algorithm model can be obtained locally or from a server. The first video frame includes a calculation frame and a warning frame. The calculation frame is an image during normal driving, and the warning frame is an image indicating an impending surge in computing power.

[0069] For example, the image of the warning frame can be an image of a sudden increase in the number of vehicles, or an image of an increase in the number of objects being analyzed, such as traffic lights, signs, and obstacles, but the present invention is not limited thereto.

[0070] In some embodiments of the present invention, the terminal can input the first video frame into a preset algorithm model for analysis and recognition processing to obtain the marking result of the first video frame; if the marking value is greater than a first preset threshold, the first video frame is a warning frame; if the marking value is not greater than the first preset threshold, the first video frame is a calculation frame.

[0071] In some embodiments of the present invention, S103 can be implemented by S1031, S1032 and S1033, as follows:

[0072] S1031. Input the first video frame into the preset algorithm model, perform analysis and recognition processing, and obtain the marking result of the first video frame.

[0073] In some embodiments of the present invention, the marking result includes a marking value, which is a percentage number.

[0074] In some embodiments of the present invention, the terminal can analyze and identify the first video frame through a preset algorithm model to obtain the marking result of the first video frame, and the marking value of the marking result is displayed in the image of the first video frame.

[0075] S1032. If the marker value is greater than the first preset threshold, then the first video frame is determined to be a warning frame.

[0076] In some embodiments of the present invention, the terminal can compare the tag value in the tagging result of the first video frame with a first preset threshold. If the tag value is greater than the first preset threshold, the first video frame is judged as a warning frame.

[0077] S1033. If the marker value is not greater than the first preset threshold, then the first video frame is determined to be the calculation frame.

[0078] In some embodiments of the present invention, the terminal can compare the tag value in the tagging result of the first video frame with a first preset threshold. If the tag value is less than the first preset threshold or the tag value is equal to the first preset threshold, the first video frame is determined to be a calculation frame.

[0079] For example, if the first preset threshold is 90%, and the label value in the labeling result of the first video frame is 92%, then the first video frame is a warning frame because the label value is greater than the first preset threshold; if the label value in the labeling result of the first video frame is 89%, then the first video frame is a calculation frame because the label value is less than the first preset threshold.

[0080] It is understood that, in some embodiments of the present invention, the terminal can input the first video frame into a preset algorithm model for analysis and recognition processing to obtain the marking result of the first video frame; if the marking value is greater than a first preset threshold, the first video frame is a warning frame; if the marking value is not greater than the first preset threshold, the first video frame is a calculation frame, providing conditions for whether to upload the first video frame to the server in the future.

[0081] S104. If the first video frame is a warning frame, then the first video frame and / or subsequent video frames of the first video frame are uploaded to the server for processing to obtain the calculation result.

[0082] In some embodiments of the present invention, the server is generally a cloud server.

[0083] In some embodiments of the present invention, if the terminal determines that the first video frame is a warning frame, it uploads the first video frame to the server. Upon receiving the first video frame, the server performs calculations to obtain a result, and finally sends the result to the terminal, which then receives the result. Alternatively, the terminal can upload subsequent video frames of the first video frame to the server. The server then performs calculations on these subsequent video frames to obtain a result, and finally sends the result to the terminal, which then receives the result. Furthermore, the terminal can upload both the first video frame and its subsequent video frames to the server. The server then performs calculations on these frames to obtain a result, and finally sends the result to the terminal. The subsequent video frames of the first video frame are further analyzed and identified using a preset algorithm model. If a subsequent video frame is a warning frame, it continues to be uploaded to the server for processing. If a subsequent video frame is a calculation frame, the upload to the server is stopped, and the calculation is performed locally.

[0084] It should be noted that the first video frame is image information acquired in real time.

[0085] In some embodiments of the present invention, S104 can be implemented by S1041 and S1042, as follows:

[0086] S1041. If the first video frame is a warning frame, establish a connection between the local machine and the server via the network.

[0087] In some embodiments of the present invention, the terminal can perform the next operation according to the type of the first video frame. If the first video frame is a warning frame, a local connection with the server is established through the network.

[0088] For example, the network is a 5G network.

[0089] S1042. Transmit the first video frame and / or subsequent video frames of the first video frame to the server for processing via the network, and obtain the calculation results from the server.

[0090] In some embodiments of the present invention, the terminal can transmit the first video frame to the server via a network. After receiving the first video frame, the server performs calculations, obtains the calculation result, and sends the calculation result to the local terminal, thereby obtaining the server's calculation result. Alternatively, the terminal can transmit subsequent video frames of the first video frame to the server via a network. The server receives the subsequent video frames of the first video frame, performs calculations, obtains the calculation result, and sends the calculation result to the local terminal, thereby obtaining the server's calculation result. Or, the terminal can transmit both the first video frame and subsequent video frames of the first video frame to the server via a network. The server receives the first video frame and subsequent video frames of the first video frame, performs calculations, obtains the calculation result, and sends the calculation result to the local terminal, thereby obtaining the server's calculation result.

[0091] For example, when the terminal determines that the first video frame is a warning frame, the vehicle will activate the 5G network, establish a connection with the server, transmit the first video frame to the server, perform calculations on it, obtain the calculation result, and then return the calculation result to the local machine; when the terminal determines that the first video frame is a warning frame, the vehicle will activate the 5G network, establish a connection with the server, transmit the first video frame and subsequent video frames to the server, perform calculations on it, obtain the calculation result, and then return the calculation result to the local machine; when the terminal determines that the first video frame is a warning frame, the vehicle will activate the 5G network, establish a connection with the server, transmit the first video frame and subsequent video frames to the server, perform calculations on it, obtain the calculation result, and then return the calculation result to the local machine.

[0092] It is understood that in some embodiments of the present invention, the terminal can perform the next operation based on the type of the first video frame. If the first video frame is an early warning frame, a local connection with the server is established through the network; the first video frame and / or subsequent video frames of the first video frame are transmitted to the server for calculation, and the calculation results of the server are obtained. The network can be opened in a timely manner to upload data to the server for calculation, thereby improving the calculation efficiency and avoiding calculation paralysis caused by sudden situations.

[0093] S105. Based on the calculation results, adjust the speed and direction during the driving process to achieve automatic driving.

[0094] In some embodiments of the present invention, the terminal adjusts the speed and direction during driving based on the calculation results to perform autonomous driving.

[0095] For example, the terminal reduces speed and moves to the right based on the calculation results.

[0096] It is understood that, in some embodiments of the present invention, the terminal can collect first video information during driving; perform decoding processing based on the first video information to obtain a first video frame; analyze and identify the type of the first video frame based on a preset algorithm model; if the first video frame is a warning frame, upload the first video frame and / or subsequent video frames of the first video frame to the server for calculation processing to obtain the calculation result; adjust the speed and direction during driving based on the calculation result to perform automatic driving, which is equivalent to pre-establishing a computing power call mechanism with the server to ensure that the vehicle can continuously maintain computing power units that meet the analysis requirements during high-speed driving, while saving bandwidth.

[0097] In some embodiments of the present invention Figure 2 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 2 ,like Figure 2 As shown, S106 and S107 are executed before S103, as follows:

[0098] S106. Obtain training sample data.

[0099] In some embodiments of the present invention, the sample data is a first warning frame; wherein, the first warning frame includes an image corresponding to the forward time point when a sudden increase in computing power occurs, and also includes an image corresponding to the forward time point when no sudden increase in computing power occurs; the image corresponding to the forward time point when a sudden increase in computing power occurs is taken as a positive sample, and the image corresponding to the forward time point when no sudden increase in computing power occurs is taken as a negative sample.

[0100] In some embodiments of the present invention, the terminal can obtain second video information and central processing unit's computational allocation record during the historical driving process; based on the computational allocation record, perform filtering processing to obtain a first computational allocation record that allocates computation to two computing units; obtain third video information corresponding to the first computational allocation record from the second video information; perform decoding processing on the third video information to obtain a third video frame, and use the third video frame as sample data.

[0101] In some embodiments of the present invention, S106 can be implemented by S1061, S1062, S1063 and S1064, as follows:

[0102] S1061. Obtain the second video information and the central processing unit's computational allocation record during the historical driving process.

[0103] In some embodiments of the present invention, the second video information is external environment information collected during the driving process; the computation allocation record includes the allocation record of computation tasks performed during the driving process.

[0104] In some embodiments of the present invention, the terminal can acquire second video information during the historical driving process through the acquisition device, and at the same time acquire the calculation allocation record of the central processing unit.

[0105] S1062. Based on the operation allocation record, perform filtering to obtain the first operation allocation record where the operation is allocated to the two computing units.

[0106] In some embodiments of the present invention, the terminal can perform filtering processing on the operation allocation record to obtain the record that allocates the operation to two computing power units, and use it as the first operation allocation record.

[0107] It should be noted that the first computation allocation record refers to the record of allocating the tasks of the main computing unit to the backup computing unit for computation processing.

[0108] S1063. Obtain the third video information corresponding to the first operation allocation record from the second video information.

[0109] In some embodiments of the present invention, the terminal can find the video information corresponding to the first operation allocation record from the second video information and use it as the third video information.

[0110] S1064. Decode the third video information to obtain the third video frame, and use the third video frame as sample data.

[0111] In some embodiments of the present invention, the terminal can obtain the third video frame corresponding to the third video information by decoding the third video information, and use the third video frame as sample data.

[0112] It is understood that, in some embodiments of the present invention, the terminal can obtain second video information and central processing unit's computational allocation record during the historical driving process; based on the computational allocation record, perform filtering processing to obtain a first computational allocation record that allocates computation to two computing units; obtain third video information corresponding to the first computational allocation record from the second video information; perform decoding processing on the third video information to obtain a third video frame, and use the third video frame as sample data to provide training sample data for obtaining a preset algorithm model.

[0113] S107. Based on the first warning frame, continuously train the initial algorithm model to determine the preset algorithm model.

[0114] In some embodiments of the present invention, the first warning frame is divided into a training warning frame and a test warning frame.

[0115] In some embodiments of the present invention, the terminal can train an initial algorithm model based on a training warning frame to obtain a training model; and perform detection and optimization on the training model based on a test warning frame to determine a preset algorithm model.

[0116] It should be noted that the preset algorithm model can be obtained from the server. The server can obtain sample data from the terminal, continuously train the initial algorithm model based on the sample data, determine the preset algorithm model, and then return the preset algorithm model to the terminal.

[0117] It is understood that, in some embodiments of the present invention, the terminal can acquire training sample data; based on the first warning frame, the initial algorithm model is continuously trained to determine the preset algorithm model, which provides a basis for judging the type of the first video frame and becomes the key to deciding whether to upload the first video frame to the server.

[0118] In some embodiments of the present invention Figure 3 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 3 ,like Figure 3 As shown, S107 can be implemented by S1071-S1072, as follows:

[0119] S1071. Based on the training warning frames, train the initial algorithm model to obtain the training model.

[0120] In some embodiments of the present invention, the terminal can train the initial algorithm model locally using training warning frames to obtain a training model; or, the terminal can send training warning frames to the server to train the initial algorithm model on the server to obtain a training model.

[0121] S1072. Based on the test warning frame, perform detection and optimization on the training model to determine the preset algorithm model.

[0122] In some embodiments of the present invention, the terminal or server can input the test warning frame into the training model to obtain the training label result of the test warning frame; if the training label value is greater than the second preset threshold, a preset algorithm model is obtained; if the training label value is not greater than the second preset threshold, training continues until the training label value in the training label result of the training model is greater than the first preset threshold, and then a preset algorithm model is obtained.

[0123] It is understood that, in some embodiments of the present invention, the terminal can train the initial algorithm model based on the training warning frame to obtain the training model; and perform detection and optimization on the training model based on the test warning frame to determine the preset algorithm model, thereby improving the recognition accuracy of the preset algorithm model.

[0124] In some embodiments of the present invention, S1072 can be implemented by S10721, S10722 and S10723, as follows:

[0125] S10721. Input the test warning frame into the training model to obtain the training labeling result of the test warning frame.

[0126] In some embodiments of the present invention, the training labeling result includes training label values.

[0127] In some embodiments of the present invention, the terminal or server can input the test warning frame into the training model, use the training model to identify the test warning frame, and obtain the training label result of the test warning frame. The training label value of the training label result is a percentage.

[0128] S10722. If the training label value is greater than the second preset threshold, the preset algorithm model is obtained.

[0129] In some embodiments of the present invention, the terminal can compare the training label value with a second preset threshold, and if the training label value is greater than the second preset threshold, output the model to obtain a preset algorithm model; or, the server can compare the training label value with the second preset threshold, and if the training label value is greater than the second preset threshold, output the model to obtain a preset algorithm model.

[0130] For example, the second preset threshold can be 90%.

[0131] S10723. If the training label value is not greater than the second preset threshold, then continue training until the training label value in the training label result of the training model is greater than the first preset threshold, and then the preset algorithm model is obtained.

[0132] In some embodiments of the present invention, the terminal can compare the training label value with a second preset threshold. If the training label value is not greater than the second preset threshold, training continues until the training label value in the training label result of the training model is greater than the second preset threshold, and then the model is output to obtain a preset algorithm model. Alternatively, the server can compare the training label value with the second preset threshold. If the training label value is not greater than the second preset threshold, training continues until the training label value in the training label result of the training model is greater than the second preset threshold, and then the model is output to obtain a preset algorithm model.

[0133] It is understood that, in some embodiments of the present invention, the terminal can input the test warning frame into the training model to obtain the training labeling result of the test warning frame; if the training labeling value is greater than the second preset threshold, a preset algorithm model is obtained; if the training labeling value is not greater than the second preset threshold, training continues until the training labeling value in the training labeling result of the training model is greater than the first preset threshold, then a preset algorithm model is obtained, and an identification standard is established, providing a basis for whether to upload the data to the server in the future.

[0134] In some embodiments of the present invention Figure 4 An optional flowchart of an autonomous driving computing power early warning method provided in an embodiment of the present invention. Figure 4 ,like Figure 4 As shown, S108 and S109 are executed after S103, as follows:

[0135] S108. If the first video frame is a calculation frame, then perform driving calculation processing locally and obtain the calculation processing load.

[0136] In some embodiments of the present invention, the terminal performs driving calculations locally based on the first video frame as the calculation frame to obtain the calculation results for autonomous driving; at the same time, it obtains the load during the calculation process.

[0137] S109. If the computational load exceeds the third preset threshold, the computational task will be assigned to the local backup computing unit for computation to obtain the computational result.

[0138] In some embodiments of the present invention, the third preset threshold is a pre-set constant.

[0139] In some embodiments of the present invention, the terminal compares the computational processing load with a third preset threshold. If the computational processing load is greater than the third preset threshold, the analysis task of the first video frame is assigned to the local backup computing unit for computation to obtain the computation result.

[0140] It should be noted that the analysis and calculation of the first video frame is mainly performed in the main computing unit of the terminal. Only when the computing power value of the main computing unit exceeds the third preset threshold will the terminal's backup computing unit be activated for calculation.

[0141] For example, the second preset threshold is 70%.

[0142] It is understood that in some embodiments of the present invention, if the terminal determines that the first video frame is a calculation frame, it will perform normal driving calculation processing to obtain the calculation processing load; if the calculation processing load is greater than a third preset threshold, the calculation task will be distributed for processing to obtain the calculation result, which can reduce the calculation load of the main computing unit and improve the calculation efficiency of the main computing unit.

[0143] The following will describe an exemplary application of the embodiments of the present invention in a practical application scenario.

[0144] Figure 5 This invention provides an optional scheme illustration of an autonomous driving computing power early warning method. Figure 1 ,like Figure 5 As shown, the autonomous driving computing power warning method mainly involves two devices: a computing power server (equivalent to a server) and a local vehicle (equivalent to a terminal). The computing power server and the local vehicle transmit data through 5G communication (equivalent to a network). The local vehicle decodes and recognizes the first video information it acquires. The recognition result is either a warning frame or a calculation frame (the calculation frame will exist throughout the entire T2 time period, while the warning frame will appear during the T1 time period, and the T2 time period is longer than the T1 time period). Based on the recognition result, it determines whether to connect and communicate with the computing power server.

[0145] An optional scheme illustration of an autonomous driving computing power early warning method provided by an embodiment of the present invention. Figure 2 ,like Figure 6 As shown, A1, A2, A3, and A4 are the main computing power unit with a maximum computing power of N, the backup computing power unit with a maximum computing power of M, the algorithm template training unit, and the recognition unit, respectively; the CPU is the central processing unit. The local vehicle's computing components include A1, A2, A3, A4, the CPU, and the FPGA. A1 is the main analysis path; A2 is the backup computing power offloading; A3 is the algorithm template training; A4 is for recognizing video information, and the recognition results are used by the CPU to determine whether to enable the 5G network and establish a communication connection; the FPGA is a pre-stored F... T1 (First warning frame), stacking is performed; video stream decoding (equivalent to decoding processing) processes the acquired first video information to obtain the first video frame. The specific process is as follows: the CPU pre-processes the F... T1 Pre-stored into the FPGA stack via a high-speed bus, for use when F T2 When the A1 computing power threshold is exceeded, F will be... T1The algorithm template is selected and sent to A3 or the computing power server for training. The trained preset algorithm template is then transmitted to A4. The first video frame (including at least one warning frame and calculation frame) is decoded and sent to A1 and A4 via the CPU for analysis and processing of the video stream. If the computing power of A1 exceeds the A1 computing power threshold, the CPU will send the first video frame to A2 for calculation. If A4 identifies the first video frame as a warning frame, the local machine will connect to the computing power server via the 5G module network, transmit the data to the computing power server for processing, and return the calculation result to the local machine for use in autonomous driving.

[0146] It is understood that, in some embodiments of the present invention, the terminal can provide early warning of computing power exhaustion by pre-judging the warning frame, thus buying time for the CPU to establish a connection and data channel with the computing power server through the 5G network. This reduces the possibility of untimely calculations due to computing power exhaustion of the vehicle's computing power unit, improves the real-time performance and reliability of external perception of vehicle operation, and maximizes the use of the vehicle's own computing power when there is no computing power exhaustion warning, reducing the consumption of 5G bandwidth resources for data exchange with cloud computing power and saving costs.

[0147] Based on the autonomous driving computing power early warning method of the above embodiments, this invention also provides an autonomous driving computing power early warning device, such as... Figure 7 The above, Figure 7 A schematic diagram of the structure of an autonomous driving computing power early warning device provided in an embodiment of the present invention. Figure 1 The device 7 includes: a data acquisition unit 701, an acquisition unit 702, a determination unit 703, and a judgment unit 704; wherein,

[0148] The acquisition unit 701 is used to acquire first video information during driving.

[0149] The acquisition unit 702 is used to perform decoding processing based on the first video information to obtain a first video frame; wherein, the first video frame is an image used for recognition and analysis;

[0150] The determining unit 703 is used to analyze and identify the type of the first video frame based on the first video frame using a preset algorithm model; wherein the preset algorithm model is a model for image judgment and recognition; the type of the first video frame includes a calculation frame and a warning frame, wherein the calculation frame is an image during normal driving, and the warning frame is an image indicating an impending surge in computing power;

[0151] The judgment unit 704 is used to upload the first video frame and / or subsequent video frames of the first video frame to the server for calculation and processing if the first video frame is a warning frame, and to obtain a calculation result; and to adjust the speed and direction during driving to perform automatic driving based on the calculation result.

[0152] In some embodiments of the present invention, the acquisition unit 702 is used to input the first video frame into the preset algorithm model for analysis and recognition processing to obtain the marking result of the first video frame; wherein, the marking result includes a marking value;

[0153] The judgment unit 704 is used to determine the first video frame as a warning frame if the mark value is greater than the first preset threshold, and to determine the first video frame as a calculation frame if the mark value is not greater than the first preset threshold.

[0154] In some embodiments of the present invention, the determination unit 704 is used to establish a local connection with the server via the network if the first video frame is a warning frame;

[0155] The acquisition unit 702 is used to transmit the first video frame and / or subsequent video frames of the first video frame to the server through the network for processing, and to obtain the calculation result of the server.

[0156] In some embodiments of the present invention, the acquisition unit 702 is used to acquire sample data for training; the sample data is a first warning frame; wherein, the first warning frame includes an image corresponding to a forward time point under the condition of a sudden increase in computing power;

[0157] The determining unit 703 is used to continuously train the initial algorithm model based on the first warning frame to determine the preset algorithm model.

[0158] In some embodiments of the present invention, the acquisition unit 702 is used to acquire second video information and central processing unit's computational allocation record during historical driving; based on the computational allocation record, perform filtering processing to obtain a first computational allocation record that allocates computation to two computing units; acquire third video information corresponding to the first computational allocation record from the second video information; decode the third video information to obtain a third video frame, and use the third video frame as the sample data.

[0159] In some embodiments of the present invention, the first warning frame is divided into a training warning frame and a test warning frame; the acquisition unit 702 is used to train the initial algorithm model according to the training warning frame to obtain a training model;

[0160] The determining unit 703 is used to perform detection optimization on the training model based on the test warning frame to determine the preset algorithm model.

[0161] In some embodiments of the present invention, the acquisition unit 702 is used to input the test warning frame into the training model to obtain the training label result of the test warning frame; the training label result includes a training label value; if the training label value is greater than a second preset threshold, the preset algorithm model is obtained; if the training label value is not greater than the second preset threshold, training continues until the training label value in the training label result of the training model is greater than a first preset threshold, and then the preset algorithm model is obtained.

[0162] In some embodiments of the present invention, the acquisition unit 702 is used to acquire the preset algorithm model from the server.

[0163] In some embodiments of the present invention, the judgment unit 704 is configured to perform driving calculation processing locally if the first video frame is a calculation frame, and obtain the calculation processing load; if the calculation processing load is greater than a third preset threshold, the calculation task is allocated to the local backup computing unit for calculation, and the calculation result is obtained.

[0164] It should be noted that the above-described division of program modules is only used as an example for autonomous driving computing power warning. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the autonomous driving computing power warning device and the autonomous driving computing power warning method provided in the above embodiments belong to the same concept. For details of their specific implementation process and beneficial effects, please refer to the method embodiments, which will not be repeated here. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiments of this invention for understanding.

[0165] Based on the autonomous driving computing power early warning method of the above embodiments, this invention also provides an autonomous driving computing power early warning device, such as... Figure 8 As shown, Figure 8 A schematic diagram of the structure of an autonomous driving computing power early warning device provided in an embodiment of the present invention. Figure 2 The device 8 includes a processor 801 and a memory 802; the memory 802 stores one or more programs executable by the processor, and when one or more programs are executed, the processor 801 executes any of the autonomous driving computing power warning methods described in the previous embodiments.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A method for early warning of computing power in autonomous driving, characterized in that, include: Collect first video information during driving; Based on the first video information, decoding processing is performed to obtain the first video frame; wherein, the first video frame is an image used for recognition and analysis; Based on the first video frame, the type of the first video frame is determined by analysis and identification using a preset algorithm model; wherein, the preset algorithm model is a model for image judgment and recognition; the type of the first video frame includes a calculation frame and a warning frame, the calculation frame is an image during normal driving, and the warning frame is an image indicating an impending surge in computing power; the preset algorithm model is obtained by training an initial algorithm model based on the first warning frame, and the first warning frame includes images corresponding to forward time points when a surge in computing power occurs; If the first video frame is a warning frame, then the first video frame and / or subsequent video frames of the first video frame are uploaded to the server for processing to obtain the calculation result. Based on the calculation results, adjust the speed and direction during driving to achieve automatic driving; Before determining the type of the first video frame by analyzing and identifying it using a preset algorithm model, the method further includes: Acquire training sample data; the sample data is the first warning frame; Based on the first warning frame, the initial algorithm model is continuously trained to determine the preset algorithm model; The acquisition of training sample data includes: Acquire second video information from the historical driving process and the central processing unit's computational allocation records; Based on the computation allocation record, a filtering process is performed to obtain the first computation allocation record where computation is allocated to two computing power units; Obtain the third video information corresponding to the first operation allocation record from the second video information; The third video information is decoded to obtain a third video frame, which is then used as the sample data.

2. The method according to claim 1, characterized in that, The step of analyzing and identifying the type of the first video frame based on the first video frame using a preset algorithm model includes: The first video frame is input into the preset algorithm model for analysis and recognition processing to obtain the labeling result of the first video frame; wherein, the labeling result includes a label value; If the marker value is greater than the first preset threshold, then the first video frame is determined to be a warning frame; If the marker value is not greater than the first preset threshold, then the first video frame is determined to be a calculation frame.

3. The method according to claim 1, characterized in that, If the first video frame is a warning frame, then the first video frame and / or subsequent video frames of the first video frame are uploaded to the server for processing to obtain the calculation result, including: If the first video frame is a warning frame, a local connection is established with the server via the network; The first video frame and / or subsequent video frames of the first video frame are transmitted to the server through the network for processing, and the calculation results of the server are obtained.

4. The method according to claim 1, characterized in that, The first warning frame is divided into training warning frames and test warning frames; The step of continuously training the initial algorithm model based on the first warning frame to determine the preset algorithm model includes: Based on the training warning frames, the initial algorithm model is trained to obtain a training model; Based on the test warning frame, the training model is tested and optimized to determine the preset algorithm model.

5. The method according to claim 4, characterized in that, The step of optimizing the trained model based on the test warning frame to determine the preset algorithm model includes: The test warning frame is input into the training model to obtain the training labeling result of the test warning frame; the training labeling result includes training labeling values; If the training label value is greater than the second preset threshold, then the preset algorithm model is obtained; If the training label value is not greater than the second preset threshold, then training continues until the training label value in the training label result of the training model is greater than the first preset threshold, and then the preset algorithm model is obtained.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the preset algorithm model from the server.

7. The method according to any one of claims 1-6, characterized in that, After determining the type of the first video frame by analyzing and identifying it using a preset algorithm model, the method further includes: If the first video frame is a calculation frame, then driving calculation processing is performed locally to obtain the calculation processing load; If the computational load exceeds a third preset threshold, the computational task will be assigned to a local backup computing unit for computation to obtain the computational result.

8. An autonomous driving computing power early warning device, characterized in that, It includes a data acquisition unit, an acquisition unit, a judgment unit, and a determination unit; among which, The acquisition unit is used to acquire first video information during driving; The acquisition unit is used to perform decoding processing based on the first video information to obtain a first video frame; wherein the first video frame is an image used for recognition and analysis; based on the first video frame, analysis and recognition are performed through a preset algorithm model to determine the type of the first video frame; wherein the preset algorithm model is a model used for image judgment and recognition; the type of the first video frame includes a calculation frame and a warning frame, wherein the calculation frame is an image during normal driving, and the warning frame is an image indicating an impending surge in computing power; the preset algorithm model is obtained by training an initial algorithm model based on the first warning frame; the first warning frame includes an image corresponding to a forward time point in the event of a surge in computing power; The judgment unit is configured to, if the first video frame is a warning frame, upload the first video frame and / or subsequent video frames of the first video frame to the server for calculation and processing to obtain a calculation result; and adjust the speed and direction during the driving process according to the calculation result to perform automatic driving. The acquisition unit is further configured to acquire training sample data; the sample data is the first warning frame; The determining unit is used to continuously train the initial algorithm model based on the first warning frame to determine the preset algorithm model; The acquisition unit is further configured to acquire second video information and central processing unit's computational allocation record during the historical driving process; perform filtering processing based on the computational allocation record to obtain a first computational allocation record that allocates computation to two computing units; acquire third video information corresponding to the first computational allocation record from the second video information; perform decoding processing on the third video information to obtain a third video frame, and use the third video frame as the sample data.

9. An autonomous driving computing power early warning device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions, which, when executed, cause the processor to perform the method as described in any one of claims 1-7.

Images