Camera video data push processing method and system, electronic device and vehicle

Abstract

The application provides a live streaming processing method and system of camera video data, an electronic device and a vehicle. The method comprises the following steps: obtaining a live streaming request instruction of a user; obtaining corresponding encoded data from a data queue based on the live streaming request instruction; the encoded data is obtained by performing hard coding processing on camera video data; in an Ffmpeg live streaming processing sub-thread, the encoded data is encapsulated into a structure object, and when it is determined that the encoded data is a key frame, a flag bit parameter corresponding to the structure object is determined to obtain a structure object to be pushed to a live streaming; and the structure object to be pushed to a live streaming is transmitted to a corresponding streaming media server. The method provided by the application can obtain camera video data in a high-performance and low-latency manner and push the camera video data to a streaming media server, thereby improving the system performance of the whole live streaming process and reducing the live streaming delay, so that the system operation resources are effectively saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video stream processing, in particular to a camera video data push stream processing method and system; in addition, an electronic device and a vehicle are also provided. BACKGROUND

[0002] In recent years, with the rapid development of network technology, data push stream technology is increasingly used in live broadcast and real-time monitoring scenarios. In the field of live broadcast or real-time monitoring, "push stream" refers to pushing video data collected from a camera to a streaming media server through a streaming media protocol (such as RTMP, RTP, RTSP, etc.), so that users can pull the video stream address on the streaming media server for playing on a browser or a mobile terminal. In the technical implementation process of push stream, a variety of open source frameworks can be used, such as Ffmpeg. Since Ffmpeg does not provide a hard coding interface, if Ffmpeg is used for the push stream scheme of encoded data, only soft coding can be used, which will occupy a large amount of CPU resources, thereby reducing the overall performance of the Android platform or Linux system.

[0003] In the prior art, the Android platform or Linux system provides a hard coding interface of MediaCodec, which is usually used to hard code the video data in the camera into H.264 encoded data, and then the encoded data is pushed to the streaming media server by Ffmpeg in RTMP protocol. However, in the Ffmpeg RTMP push stream process, the existing RTMP push stream process is complicated, and has defects such as poor performance, high latency, etc. If not handled properly, it may cause problems such as failure to pull stream and play, large playback delay, or low system performance for push stream, etc. Therefore, how to design a camera video data push stream processing scheme that can effectively improve the system performance and reduce the push stream latency has become a technical problem to be solved. SUMMARY

[0004] To this end, the present application provides a camera video data push stream processing method to solve the defects of high limitation of the existing camera video data push stream processing scheme, poor system performance and push stream latency.

[0005] The present application provides a camera video data push stream processing method, comprising:

[0006] obtaining a push stream request instruction of a user;

[0007] obtaining corresponding encoded data from a data queue based on the push stream request instruction; the encoded data is obtained by hard coding the camera video data;

[0008] In the Ffmpeg streaming processing sub-thread, the encoded data is encapsulated into a structure object, and when it is determined that the encoded data is a key frame, a flag bit parameter corresponding to the structure object is determined to obtain a structure object to be streamed.

[0009] The structure object to be streamed is transmitted to a corresponding streaming media server.

[0010] Further, the camera video data streaming processing method further comprises: after transmitting the structure object to be streamed to the corresponding streaming media server, performing a time delay operation on the Ffmpeg streaming processing sub-thread.

[0011] Further, the time delay operation on the Ffmpeg streaming processing sub-thread specifically comprises:

[0012] Obtaining a first current timestamp information corresponding to the transmission of the structure object;

[0013] Based on the first current timestamp information, the initial timestamp information, and the preset time delay model, a corresponding time difference is determined; wherein the initial timestamp information is the timestamp information corresponding to the camera video data;

[0014] Based on the time difference, the Ffmpeg streaming processing sub-thread is time delayed.

[0015] Further, the camera video data streaming processing method further comprises: after obtaining the user's streaming request instruction, when the data queue is empty, performing a time delay operation on the Ffmpeg streaming processing sub-thread.

[0016] Further, when the data queue is empty, the Ffmpeg streaming processing sub-thread is time delayed, specifically comprising: during the running of the Ffmpeg streaming processing sub-thread, when it is determined that there is no encoded data in the data queue, a corresponding time delay link is entered, and based on the obtained second current timestamp information, the initial timestamp information, and the preset time delay model, a corresponding time difference is determined; wherein the second current timestamp information is the timestamp information corresponding to the determination that the data queue is empty, and the initial timestamp information is the timestamp information corresponding to the camera video data;

[0017] Based on the time difference, the streaming processing sub-thread is time delayed to release the corresponding system operation resources during the encoding data acquisition process.

[0018] Further, before obtaining the user's streaming request instruction, it further comprises:

[0019] acquire the camera video data;

[0020] perform hard coding processing on the camera video data to obtain corresponding coding data;

[0021] store the coding data to a preset data queue; wherein the data queue is a blocking queue.

[0022] Further, when the coding data is determined to be a key frame, the flag parameter corresponding to the structure object is determined to obtain the structure object to be pushed, specifically comprising: when the coding data is determined to be a key frame, the flag parameter corresponding to the structure object is determined to be AV_PKT_FLAG_KEY; wherein the key frame includes a sequence parameter set, an image parameter set and an instant decoder refresh data.

[0023] The application further provides a camera video data pushing processing system, comprising:

[0024] a request instruction obtaining module for obtaining a pushing request instruction of a user;

[0025] a coding data obtaining module for obtaining corresponding coding data from a data queue based on the pushing request instruction; the coding data is obtained by performing hard coding processing on camera video data;

[0026] an Ffmpeg pushing module for encapsulating the coding data into a structure object in an Ffmpeg pushing processing sub-thread, and determining a flag parameter corresponding to the structure object when the coding data is determined to be a key frame, so as to obtain a structure object to be pushed;

[0027] a data transmission module for transmitting the structure object to be pushed to a corresponding streaming media server.

[0028] Further, the camera video data pushing processing system further comprises a first delay processing module for performing a delay operation on the Ffmpeg pushing processing sub-thread after the structure object to be pushed is transmitted to the corresponding streaming media server.

[0029] Further, the delay operation on the Ffmpeg pushing processing sub-thread specifically comprises:

[0030] acquiring a first current timestamp information corresponding to the transmission of the structure object;

[0031] determining a corresponding time difference based on the first current timestamp information, initial timestamp information and a preset delay model; wherein the initial timestamp information is a timestamp information corresponding to the acquisition of camera video data;

[0032] The Ffmpeg live streaming processing sub-thread is delayed based on the time difference.

[0033] Further, the camera video data live streaming processing system further comprises a second delay processing module configured to, after obtaining a user live streaming request instruction, delay the Ffmpeg live streaming processing sub-thread when the data queue is empty.

[0034] Further, the delaying of the Ffmpeg live streaming processing sub-thread when the data queue is empty specifically comprises:

[0035] During the running of the Ffmpeg live streaming processing sub-thread, when it is determined that the encoding data does not exist in the data queue, a corresponding delay link is entered, and a corresponding time difference is determined based on the obtained second current timestamp information, initial timestamp information and a preset delay model, wherein the second current timestamp information is the timestamp information corresponding to the determination that the data queue is empty, and the initial timestamp information is the timestamp information corresponding to the camera video data.

[0036] The Ffmpeg live streaming processing sub-thread is delayed based on the time difference to release corresponding system operation resources in the encoding data acquisition process.

[0037] Further, before obtaining the user live streaming request instruction, the method further comprises:

[0038] The video data acquisition module is configured to acquire the camera video data.

[0039] The encoding processing module is configured to perform hard encoding processing on the camera video data to obtain corresponding encoding data.

[0040] The encoding data storage module is configured to store the encoding data in a preset data queue, wherein the data queue is a blocking queue.

[0041] Further, the Ffmpeg live streaming module is specifically configured to, when it is determined that the encoding data is a key frame, determine that the corresponding flag parameter in the structure object is AV_PKT_FLAG_KEY, wherein the key frame comprises a sequence parameter set, a picture parameter set and an instant decoder refresh data.

[0042] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the steps of the camera video data live streaming processing method according to any one of the above embodiments when executing the program.

[0043] The application also provides a vehicle comprising the electronic device.

[0044] The application provides a push stream processing method for camera video data, which comprises the following steps: obtaining a push stream request instruction, acquiring corresponding encoded data from a data queue, encapsulating the structure encoded data into a structure object in a push stream processing sub-thread, and directly determining a flag bit parameter corresponding to the structure object after determining that the encoded data is a key frame, so as to obtain an object to be pushed; and then transmitting the structure object to be pushed to a corresponding stream media server. The method can acquire camera video data in a high-performance and low-delay manner, and push the camera video data to the stream media server, thereby improving the system performance of the whole push stream process and reducing the push stream delay, so that the system operation resources are effectively saved. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0046] Figure 1 FIG. 1 is a flowchart of a push stream processing method for camera video data provided by the application;

[0047] Figure 2 FIG. 2 is a complete flowchart of the push stream processing method for camera video data provided by the application;

[0048] Figure 3 FIG. 3 is a logic flowchart of an OnPreviewFrame callback interface provided by the application;

[0049] Figure 4 FIG. 4 is a structure diagram of a push stream processing system for camera video data provided by the application;

[0050] Figure 5 FIG. 5 is a specific implementation framework diagram of the push stream processing method for camera video data provided by the application;

[0051] Figure 6 FIG. 6 is a structure diagram of an electronic device provided by the application. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0053] The embodiments of the push streaming processing method of camera video data according to the present application will be described in detail below. As shown in Figure 1 The flowchart of the push streaming processing method of camera video data provided by the embodiments of the present application is shown in

[0054] Step 101: Obtain the push request instruction of the user.

[0055] In the specific implementation process of the present application, if the push request instruction of the user is detected, such as the live viewing instruction of the specified camera input by the user based on the Android device, the scalar isPush is set to true (the initial default value of isPush is false), and then Ffmpeg is initialized. Specifically, the initialization actions include a series of conventional actions such as av_register_all(), avformat_network_init(), avformat_new_stream(), av_dump_format() and avformat_write_header(). If the initialization fails, such as the initialization failure caused by the network disconnection, the communication failure between the Android device and the streaming media server, etc., the Ffmpeg push streaming processing is not allowed to be performed. If the initialization is completed, a processing sub-thread thread1 for Ffmpeg push streaming, i.e., the Ffmpeg push streaming processing sub-thread, is newly created.

[0056] It should be noted that before receiving the user's streaming request instruction, the process also includes: acquiring the camera video data, performing hard encoding on the camera video data to obtain corresponding encoded data, and then storing the encoded data in a preset data queue; wherein, the data queue is a blocking queue. For example, by utilizing the MediaCodec hard encoding interface capability of the Android platform, a data queue (i.e., a blocking queue ArrayBlockingQueue) can be designed. After the camera video data is encoded into encoded data (such as H.264 or H.265 encoded data) through the MediaCodec hard encoding interface, the encoded data is transmitted to Ffmpeg, thereby improving the overall performance of the Android system when acquiring encoded data during Ffmpeg streaming.

[0057] Specifically, after initializing the relevant camera parameters (such as setting resolution, data format, frame rate, etc.), video data from the specified camera can be acquired in real time. Then, the corresponding camera video data is obtained based on the OnPreviewFrame callback interface provided by the Android system. After initializing the MediaCodec hard encoding interface, the acquired camera video data is hard encoded using this MediaCodec hard encoding interface. Specifically, the OnPreviewFrame callback interface executes the following... Figure 3 The processing flow is as follows: Step 302, first determine if there is encoded data in OnPreviewFrame. If so, proceed to step 303 to obtain camera video data. Upon obtaining camera video data, proceed to step 304 to record the current system timestamp corresponding to the obtained camera video data. Use the current system timestamp as the initial timestamp information (defined as T1). Then, proceed to step 305 to implement hard encoding processing of the camera video data using the MediaCodec hard encoding interface provided by the Android system. In the implementation of the OnPreviewFrame callback interface, this invention sets up a corresponding data queue (i.e., a blocking queue ArrayBlockingQueue, which can be defined as dataQueue). After obtaining the encoded data obtained through MediaCodec hard encoding, proceed to step 306 to store the encoded data in dataQueue. In the specific implementation of this invention, when obtaining the camera video data based on the OnPreviewFrame callback interface, the current initial timestamp information is obtained accordingly; then, the camera video data is hard encoded based on the MediaCodec hard encoding interface to obtain the corresponding encoded data; finally, the encoded data is stored in the corresponding data queue.

[0058] Step 102: obtaining corresponding encoded data from the data queue based on the push request instruction; the encoded data is obtained by hard encoding the camera video data.

[0059] Specifically, the data queue is a preset blocking queue ArrayBlockingQueue. The encoded data can be H.264 encoded data or H.265 encoded data. Figure 2 As shown in the figure, after obtaining the push request instruction, it is necessary to first execute step 2061 to judge whether isPush is true. If not, it indicates that the current push processing is not needed to continue. If yes, it indicates that the current push processing is needed to continue. It is also necessary to further execute step 2062 to judge whether the data queue (i.e. dataQueue) is empty. If not (i.e. there is encoded data in the data queue), step 2063 is executed to obtain corresponding encoded data from the data queue dataQueue. If yes (i.e. there is no encoded data in the data queue), step 2010 is executed to obtain the current timestamp T3 of the system and perform corresponding delay operation. It should be noted that the principle of obtaining encoded data from the data queue dataQueue is based on the first-in first-out principle. According to the principle, the encoded data obtained each time is the encoded data stored before.

[0060] Step 103: in the Ffmpeg push processing sub-thread, the encoded data is encapsulated into a structure object, and when it is judged that the encoded data is a key frame, a flag parameter corresponding to the structure object is determined, so as to obtain a structure object to be pushed.

[0061] The structure object refers to an AVPacket object in the Ffmpeg push processing sub-thread. The flag parameter is a flags parameter of the AVPacket object.

[0062] In the embodiment of the present application, the encoded data can be assigned to the data member variable of the AVPacket object in the Ffmpeg live streaming processing sub-thread through the JNI (Java Native Interface) technology. In the process of encapsulating the AVPacket object, the flag parameter in the AVPacket object needs to be accurately set. If the setting is not accurate, the live streaming playback will fail or be stuck, so it is necessary to first determine whether the encoded data (such as H.264 or H.265 encoded data) is a key frame. If yes, the flags parameter of the AVPacket object is determined when the encoded data is determined to be a key frame to obtain the AVPacket object to be pushed. The key frame includes sequence parameter set, picture parameter set and instantaneous decoder refresh data. Specifically, when the encoded data is determined to belong to sequence parameter set (SPS, Sequence Parameter Set), picture parameter set (PPS, Picture Parameter Set) and instantaneous decoder refresh data (IDR, Instantaneous Decoding Refresh) key frame, the corresponding flags parameter in the AVPacket object is set to AV_PKT_FLAG_KEY. If not, it is set to 0 (of course, it can also be set to other different values). It should be noted that in the process of encapsulating the AVPacket object, the initial default value of the flags parameter can be set to 0. If the value is AV_PKT_FLAG_KEY, it means that the data member variable data in the current AVPacket object is a key frame. If the value is not actively set to AV_PKT_FLAG_KEY at the corresponding time point (such as when the encoded data is determined to belong to SPS or PPS or IDR data), the value will always remain the initial default value 0. If the value is not set to AV_PKT_FLAG_KEY when the encoded data is determined to belong to SPS or PPS or IDR data, the live streaming playback will fail or be stuck.

[0063] Step 104: transmitting the structure object to be pushed to the corresponding streaming media server.

[0064] In the embodiment of the present application, the AVPacket object to be pushed can be transmitted to the streaming media server based on the av_interleaved_write_frame interface of Ffmpeg. If the interface returns 0, it means that the live streaming is successful, at this time the video stream on the streaming media server can be pulled and played through the player, that is, the video content of the monitoring camera can be seen through the player, thereby greatly improving the performance of the live streaming and the live streaming delay is in the order of milliseconds.

[0065] Specifically, after the structure object to be pushed to the stream is pushed to the corresponding stream media server, the Ffmpeg pushing process sub-thread can be delayed, so that the CPU resources of the system can be released to other logical execution opportunities that need CPU resources at appropriate times to improve the performance of the system. The corresponding implementation process of delaying the Ffmpeg pushing process sub-thread includes: obtaining a first current timestamp information corresponding to the transmission of the structure object; determining a corresponding time difference based on the first current timestamp information, initial timestamp information, and a preset delay model. The initial timestamp information is the timestamp information corresponding to the acquisition of the camera video data; based on the time difference, the Ffmpeg pushing process sub-thread is subjected to corresponding delay operation to release the corresponding system operation resources in the data pushing process. That is, after the AVPacket object to be pushed is transmitted to the corresponding stream media server, the first current timestamp information corresponding to the transmission of the AVPacket object is obtained; based on the first current timestamp information, the initial timestamp information, and the preset delay model, a corresponding time difference is determined; based on the time difference, the Ffmpeg pushing process sub-thread is subjected to corresponding delay operation to release the corresponding system operation resources in the data pushing process. The delay model (delay Sleep) corresponds to the calculation formula:

[0066] Time difference = 1000 / frame number-(T2-T1) (1)

[0067] Wherein, T2 in the delay model is the first current timestamp information; T1 is the initial timestamp information; the frame number is a parameter determined by initializing the camera; 1000 is a preset parameter, which can also be set to 2000, and can be selected as needed, and is not limited here.

[0068] If the time difference T2-T1 is less than or equal to 0, Thread.sleep(0) is called; otherwise, Thread.sleep(time difference) is called. Thread.sleep() is a function of calling delay operation, and the delay time is the time in the parentheses, such as 0 or time difference. The delay time and the calculated time difference can be equal or have a certain corresponding relationship, which is not limited here.

[0069] As Figure 2As shown, the application reorganizes the existing Ffmpeg RTMP push stream process and formulates the Ffmpeg RTMP push stream process. After Ffmpeg receives the encoded data, the encoded data is pushed by the Ffmpeg RTMP (Real Time Messaging Protocol) push stream process to solve the defect that Ffmpeg cannot realize high-performance RTMP push stream on the Android system due to the lack of support for hardware encoding, improve the performance of RTMP push stream processing, reduce the delay of Ffmpeg push stream, and enable Ffmpeg to push the camera video data obtained on the Android system to the streaming media server in the RTMP protocol mode with high performance and low delay, and then pull stream to realize real-time monitoring and live broadcast functions.

[0070] In the processing logic of the newly created sub-thread thread1 (i.e. the Ffmpeg push stream processing sub-thread), after step 2061 is executed to determine that isPush is true, step 2062 is further executed to determine whether dataQueue is empty. If it is empty, it indicates that there is no encoded data in the data queue that needs to be pushed, and the corresponding delay link is entered, i.e. step 2010 is executed to obtain the current timestamp T3 of the system, and step 2011 is executed to delay the Ffmpeg push stream processing sub-thread, thereby improving the performance of the system. In the implementation process of the application, an infinite loop process is running in the Ffmpeg push stream processing sub-thread. As long as the value of isPush is true, the loop process will continue to run, and a corresponding delay operation will be performed once in each loop process, so that the CPU resources of the system can be released at the corresponding time, and opportunities for other logic execution that needs CPU resources can be provided, so as to improve the performance of the system. When the dataQueue is empty, the Ffmpeg push stream processing sub-thread is delayed, and the corresponding specific implementation process includes: after obtaining the user's push stream request instruction, when there is no encoded data in the data queue during the running process of the Ffmpeg push stream processing sub-thread, the corresponding delay link is controlled to enter, and based on the obtained second current timestamp information, initial timestamp information and delay model, the corresponding time difference is determined; based on the time difference, the Ffmpeg push stream processing sub-thread is delayed to release the corresponding system operation resources during the encoded data acquisition process. Wherein, the second current timestamp information is the timestamp information corresponding to the determination that the data queue is empty, and the initial timestamp information is the timestamp information corresponding to the acquisition of the camera video data. The calculation formula of the delay model is:

[0071] Time difference = 1000 / frame number - (T3-T1) (2)

[0072] T3 = T1 + frame * 1000 wherein T3 in the delay model is second current timestamp information; T1 is initial timestamp information; frame number is a parameter determined by initializing the camera; and 1000 is a preset parameter, which is not specifically limited here.

[0073] Correspondingly, if the time difference is less than or equal to 0, Thread.sleep(0) is called; otherwise, Thread.sleep(time difference) is called. Thread.sleep() is a function for calling a delay operation, and the delay time is the time in the parentheses, such as 0 or the time difference. The delay time and the calculated time difference can be equal or have a certain corresponding relationship, which is not specifically limited here.

[0074] In addition, if the dataQueue is not empty, step 2063 is performed to take out the encoded data from the data queue, and then step 2064 is performed to assign the encoded data to the data member variable in the AVPacket object of Ffmpeg through the technology of JNI. After the encoded data is pushed out by Ffmpeg, the sub-thread thread1 also performs a corresponding delay operation, so as to release the CPU resources of the system at the appropriate time, give other logical execution opportunities that need CPU resources, and improve the performance of the system. Specifically, in the loop sub-thread of obtaining the encoded data in the data queue and pushing the stream by Ffmpeg, after traversing the data queue and processing each frame of encoded data, a delay operation in accordance with a specific delay model is performed. First, step 2065 is performed to determine whether the encoded data is a key frame. If not, step 2067 is performed to set the flags of the AVPacket object to a preset parameter (such as 0 or 1). If yes, step 2066 is performed to set the flags of the AVPacket object to AV_PKT_FLAG_KEY. Further, step 2068 is performed to push the data to the streaming media server and obtain the current timestamp T3 of the system, and perform a corresponding delay operation. This process improves the overall performance of the Android platform in obtaining encoded data during the streaming process, including reducing the CPU usage; by immediately setting the flags parameter of the AVPacket object after determining the key frame of the encoded data, and then formally pushing the stream, the Ffmpeg streaming process is simplified, and the streaming delay is reduced.

[0075] In summary, the camera video data push processing method provided by the embodiment of the application obtains a push request instruction, obtains corresponding encoded data from a data queue, encapsulates the encoded data into a structure object in a push processing sub-thread, and directly determines a flag bit parameter corresponding to the structure object after determining that the encoded data is a key frame, to obtain a structure object to be pushed; and then the structure object to be pushed is transmitted to a corresponding streaming media server. This enables Ffmpeg to obtain camera video data in a high-performance and low-latency manner, and push the camera video data to the streaming media server, thereby improving the system performance of the entire push process and reducing the push delay, so that system operation resources are effectively saved.

[0076] Corresponding to the camera video data push processing method provided above, the application further provides a camera video data push processing system. Since the embodiment of the system is similar to the above-mentioned method embodiment, it is described relatively simply, and the relevant parts can be referred to the above-mentioned method embodiment part. The embodiment of the camera video data push processing system described below is only illustrative. Please refer to Figure 4 Fig. 1 shows a structure diagram of a camera video data push processing system provided by an embodiment of the application.

[0077] The camera video data push processing system provided by the application specifically comprises:

[0078] The request instruction obtaining module 401 is configured to obtain a push request instruction of a user;

[0079] The encoded data obtaining module 402 is configured to obtain corresponding encoded data from a data queue based on the push request instruction; the encoded data is obtained by hard encoding processing of camera video data;

[0080] The Ffmpeg push module 403 is configured to encapsulate the encoded data into a structure object in an Ffmpeg push processing sub-thread, and determine a flag bit parameter corresponding to the structure object when it is determined that the encoded data is a key frame, to obtain a structure object to be pushed;

[0081] The data transmission module 404 is configured to transmit the structure object to be pushed to a corresponding streaming media server.

[0082] Further, the camera video data push processing system further comprises a first delay processing module configured to perform a delay operation on the Ffmpeg push processing sub-thread after the structure object to be pushed is transmitted to the corresponding streaming media server.

[0083] The delay operation on the Ffmpeg stream processing sub-thread specifically includes: obtaining a first current timestamp information corresponding to the structure object; determining a corresponding time difference based on the first current timestamp information, initial timestamp information, and a preset delay model; wherein the initial timestamp information is a timestamp information corresponding to the camera video data; and performing a delay operation on the Ffmpeg stream processing sub-thread based on the time difference.

[0084] Further, the camera video data stream processing system further includes: a second delay processing module configured to perform a delay operation on the Ffmpeg stream processing sub-thread when the data queue is empty after obtaining a stream request instruction of a user.

[0085] The delay operation on the Ffmpeg stream processing sub-thread when the data queue is empty specifically includes: when it is determined that there is no encoding data in the data queue during the running of the Ffmpeg stream processing sub-thread, entering a corresponding delay link, and determining a corresponding time difference based on the obtained second current timestamp information, initial timestamp information, and a preset delay model; wherein the second current timestamp information is a timestamp information corresponding to when it is determined that the data queue is empty, and the initial timestamp information is a timestamp information corresponding to the camera video data.

[0086] The delay operation on the Ffmpeg stream processing sub-thread based on the time difference is performed to release corresponding system operation resources during the acquisition of the encoding data.

[0087] Further, before obtaining the stream request instruction of the user, the system further includes:

[0088] A video data acquisition module configured to acquire the camera video data.

[0089] An encoding processing module configured to perform a hard encoding processing on the camera video data to obtain corresponding encoding data.

[0090] An encoding data storage module configured to store the encoding data in a preset data queue; wherein the data queue is a blocking queue.

[0091] Further, the Ffmpeg stream module is specifically configured to: when it is determined that the encoding data is a key frame, determine that a corresponding flag parameter in the structure object is AV_PKT_FLAG_KEY; wherein the key frame includes a sequence parameter set, a picture parameter set, and an instant decoder refresh data.

[0092] As Figure 5As shown, the specific application process includes a device based on an Android or Linux system, a streaming media server 407, and a streaming playback terminal 408. The device includes a request instruction acquisition module 401, an encoded data acquisition module 402, a camera data acquisition module 405, a MediaCodec hardware encoding module 406, a data transmission module 404, and an Ffmpeg push streaming module 403, which are the core modules of this invention. The streaming media server 407 receives RTMP push streaming data from the device, so that the streaming playback terminal 408 can pull the data for playback. The streaming playback terminal 408 is used to pull video streams from the streaming media server for playback.

[0093] The camera video data streaming processing system described in this embodiment of the invention obtains the corresponding encoded data from the data queue upon receiving a streaming request instruction. In the streaming processing sub-thread, the encoded data is encapsulated into a structure object. After determining that the encoded data is a keyframe, the system directly determines the flag parameters corresponding to the structure object to obtain the structure object to be streamed. Then, the structure object to be streamed is transmitted to the corresponding streaming media server. This enables FFmpeg to obtain camera video data in a high-performance and low-latency manner and push it to the streaming media server, improving the system performance and reducing streaming latency throughout the streaming process, thereby effectively saving system computing resources.

[0094] Corresponding to the above-described method for streaming camera video data, this invention also provides an electronic device. Since the embodiment of this electronic device is similar to the above-described method embodiment, it is described simply. For relevant details, please refer to the description in the above-described method embodiment section. The electronic device described below is merely illustrative. Figure 6 The diagram shown is a schematic representation of the physical structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may include a processor 601, a memory 602, and a communication bus 603. The processor 601 and the memory 602 communicate with each other via the communication bus 603 and communicate with external systems via a communication interface 604. The processor 601 can call logical instructions in the memory 602 to execute a streaming processing method for camera video data. This method includes: obtaining a user's streaming request instruction; obtaining corresponding encoded data from a data queue based on the streaming request instruction; the encoded data being obtained by hard-coding camera video data; in an Ffmpeg streaming processing sub-thread, encapsulating the encoded data into a structure object, and determining the flag parameters corresponding to the structure object when the encoded data is determined to be a keyframe, to obtain the structure object to be streamed; and transmitting the structure object to be streamed to a corresponding streaming media server.

[0095] Further, the logic instructions in the memory 602 described above can be implemented in the form of software function modules and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0096] In another aspect, the present application also provides a vehicle, comprising: the electronic device. The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the above-mentioned camera video data streaming processing method is realized, which comprises: obtaining a user's streaming request instruction; obtaining corresponding encoded data from the data queue based on the streaming request instruction; the encoded data is obtained by hard coding processing of camera video data; in the Ffmpeg streaming processing sub-thread, the encoded data is encapsulated into a structure object, and when it is determined that the encoded data is a key frame, a flag parameter corresponding to the structure object is determined to obtain a structure object to be streamed; and the structure object to be streamed is transmitted to a corresponding streaming media server.

[0097] The above-described embodiments are merely illustrative, and the modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, i.e. they can be located in one place or distributed on multiple network modules. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0098] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium of a vehicle, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device of the vehicle (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some part of the embodiment.

[0099] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for streaming and processing camera video data, characterized in that, include: Obtain the user's push stream request instruction; Based on the push request instruction, retrieve the corresponding encoded data from the data queue; The encoded data is obtained by hard-coding the camera video data; When the data queue is empty, the Ffmpeg push stream processing sub-thread is delayed. During the execution of the Ffmpeg push stream processing sub-thread, when it is determined that there is no encoded data in the data queue, a corresponding delay phase is entered, and the corresponding time difference is determined based on the acquired second current timestamp information, initial timestamp information, and preset delay model; wherein, the second current timestamp information is the timestamp information corresponding to the determination that the data queue is empty, and the initial timestamp information is the timestamp information corresponding to the acquisition of camera video data; Based on the time difference, the Ffmpeg push stream processing sub-thread is delayed; In the Ffmpeg streaming sub-thread, the encoded data is encapsulated into a structure object, and when the encoded data is determined to be a keyframe, the flag parameter corresponding to the structure object is determined to obtain the structure object to be streamed. The structure object refers to the AVPacket object in the Ffmpeg streaming sub-thread, and the flag parameter is the flags parameter of the AVPacket object. The structure object to be streamed is transmitted to the corresponding streaming media server.

2. The method for streaming and processing camera video data according to claim 1, characterized in that, After transmitting the structure object to be streamed to the corresponding streaming media server, the method further includes: performing a delay operation on the Ffmpeg streaming processing sub-thread.

3. The method for streaming and processing camera video data according to claim 2, characterized in that, The delay operation on the Ffmpeg push stream processing sub-thread specifically includes: obtaining the first current timestamp information corresponding to the transmitted structure object; Based on the first current timestamp information, the initial timestamp information, and the preset delay model, the corresponding time difference is determined; wherein, the initial timestamp information is the timestamp information corresponding to the acquisition of camera video data; The Ffmpeg push stream processing sub-thread is delayed based on the time difference.

4. The method for streaming and processing camera video data according to claim 1, characterized in that, Before receiving the user's push stream request instruction, it also includes: Acquire the video data from the camera; The camera video data is hard-coded to obtain the corresponding encoded data; The encoded data is stored in a preset data queue; wherein the data queue is a blocking queue.

5. The method for streaming and processing camera video data according to claim 1, characterized in that, When determining that the encoded data is a keyframe, the step of determining the flag parameters corresponding to the structure object to obtain the structure object to be streamed specifically includes: When determining that the encoded data is a keyframe, the corresponding flag parameter in the structure object is determined to be AV_PKT_FLAG_KEY; wherein, the keyframe includes a sequence parameter set, an image parameter set, and real-time decoder refresh data.

6. A system for streaming and processing camera video data, characterized in that, include: The request instruction acquisition module is used to obtain the user's push stream request instruction; The encoded data acquisition module is used to acquire corresponding encoded data from the data queue based on the push request instruction; the encoded data is obtained by hard encoding the camera video data. When the data queue is empty, the Ffmpeg push stream processing sub-thread is delayed. During the execution of the Ffmpeg push stream processing sub-thread, when it is determined that there is no encoded data in the data queue, a corresponding delay phase is entered, and the corresponding time difference is determined based on the acquired second current timestamp information, initial timestamp information, and preset delay model; wherein, the second current timestamp information is the timestamp information corresponding to the determination that the data queue is empty, and the initial timestamp information is the timestamp information corresponding to the acquisition of camera video data; Based on the time difference, the Ffmpeg push stream processing sub-thread is delayed; The Ffmpeg streaming module is used to encapsulate the encoded data into a structure object in the Ffmpeg streaming processing sub-thread, and determine the flag parameters corresponding to the structure object when the encoded data is determined to be a keyframe, so as to obtain the structure object to be streamed. The structure object refers to the AVPacket object in the Ffmpeg streaming processing sub-thread, and the flag parameters are the flags parameters of the AVPacket object. The data transmission module is used to transmit the structure object to be streamed to the corresponding streaming media server.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the camera video data streaming processing method as described in any one of claims 1 to 5.

8. A vehicle, characterized in that, include: The electronic device as claimed in claim 7.

Images