Audio and video playing method, device, equipment and medium

Abstract

The present disclosure provides an audio and video playing method and device, equipment and medium, relates to the field of media data processing, and particularly to the field of intelligent transportation and automatic driving. The specific implementation scheme is: receiving and buffering audio data and video data; obtaining the audio data and the video data and playing synchronously; adding a control icon in the played video image, the control icon being used for receiving a playing control instruction to control audio and video playing according to the playing control instruction. The embodiment of the present disclosure can realize synchronous playing and improve control real-time performance.

Description

Technical Field

[0001] This disclosure relates to the field of media data processing, specifically to the fields of intelligent transportation and autonomous driving, and particularly to an audio and video playback method, apparatus, device, and medium. Background Technology

[0002] Currently, the biggest demand in the in-vehicle scenario is the mobile phone screen mirroring function, which projects the content of the mobile phone onto the car's infotainment system, enriching the ecosystem of the car's infotainment system.

[0003] When playing audio and video data on terminal devices or televisions, the video and sound need to be synchronized to ensure a good viewing experience for the user. Summary of the Invention

[0004] This disclosure provides an audio and video playback method, apparatus, device, and medium.

[0005] According to one aspect of this disclosure, an audio / video playback method is provided, comprising:

[0006] Receive and cache audio and video data;

[0007] Acquire the audio data and the video data, and play them synchronously;

[0008] Add a control icon to the playing video image. The control icon is used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0009] According to another aspect of this disclosure, an audio / video playback method is provided, comprising:

[0010] Acquire video and audio data;

[0011] The video data and audio data are sent to the controlled terminal so that the controlled terminal receives and caches the audio data and video data; the audio data and video data are acquired and played synchronously; a control icon is added to the playing video image, the control icon being used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0012] According to one aspect of this disclosure, an audio / video playback device is provided, comprising:

[0013] The data receiving module is used to receive and buffer audio and video data;

[0014] A synchronized playback module is used to acquire the audio data and the video data and play them synchronously.

[0015] The control icon adding module is used to add control icons to the video image being played. The control icons are used to receive playback control commands to control the audio and video playback according to the playback control commands.

[0016] According to one aspect of this disclosure, a training device for an audio / video playback module is provided, comprising:

[0017] The source data acquisition module is used to acquire video and audio data;

[0018] The data transmission module is used to send the video data and the audio data to the controlled terminal, so that the controlled terminal receives and buffers the audio data and video data; acquire the audio data and the video data and play them synchronously; add a control icon to the video image being played, the control icon being used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0020] At least one processor; and

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the audio and video playback method according to any embodiment of this disclosure.

[0023] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the audio and video playback method described in any embodiment of this disclosure.

[0024] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the audio and video playback method described in any embodiment of this disclosure.

[0025] The embodiments disclosed herein can achieve synchronized playback and improve real-time control.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0027] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0028] Figure 1 This is a flowchart of an audio / video playback method disclosed in an embodiment of the present disclosure;

[0029] Figure 2 This is a flowchart of another audio / video playback method disclosed in the embodiments of this disclosure;

[0030] Figure 3 This is a flowchart of another audio / video playback method disclosed in the embodiments of this disclosure;

[0031] Figure 4 This is a flowchart of an audio / video playback method disclosed in an embodiment of the present disclosure;

[0032] Figure 5 This is a scene diagram of an audio / video playback method disclosed in an embodiment of this disclosure;

[0033] Figure 6 This is a schematic diagram illustrating the data transmission between a mobile phone and an in-vehicle terminal according to an embodiment of this disclosure;

[0034] Figure 7 This is a schematic diagram illustrating the receiving and processing of data by an in-vehicle terminal according to an embodiment of this disclosure;

[0035] Figure 8 This is a schematic diagram illustrating the receiving and processing of data by an in-vehicle terminal according to an embodiment of this disclosure;

[0036] Figure 9 This is a schematic diagram of a control area identification according to an embodiment of the present disclosure;

[0037] Figure 10 This is a schematic diagram of a video image disclosed according to an embodiment of the present disclosure;

[0038] Figure 11 This is a schematic diagram of a video control system disclosed in an embodiment of the present disclosure;

[0039] Figure 12 This is a schematic diagram of the structure of an audio / video playback device disclosed in an embodiment of the present disclosure;

[0040] Figure 13 This is a schematic diagram of the structure of an audio / video playback device disclosed in an embodiment of the present disclosure;

[0041] Figure 14 This is a block diagram of an electronic device used to implement the audio and video playback method of the embodiments of this disclosure. Detailed Implementation

[0042] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0043] Figure 1 This is a flowchart of an audio / video playback method disclosed in an embodiment of the present disclosure. This embodiment can be applied to the synchronous playback of audio and video data of the same video. The method of this embodiment can be executed by an audio / video playback device, which can be implemented in software and / or hardware and specifically configured in an electronic device with a certain data processing capability. The electronic device can be a client device or a server device. Client devices include mobile phones, tablet computers, vehicle terminals, and desktop computers.

[0044] S101, Receive and buffer audio and video data.

[0045] The current electronic device receives audio and video data from the data source. The audio and video data originate from the same video file. When this video file is played, the audio and video data need to be played synchronously. The current electronic device receives the audio and video data and buffers them.

[0046] Optionally, receiving audio and video data includes: acquiring audio and video data sent by the data source end in the same communication channel.

[0047] In this process, the electronic device establishes a communication channel with the data source to transmit audio and video data. For example, the RTMP (Real Time Messaging Protocol) can be used to transmit audio and video data. Typically, the amount of audio data within the same time period is less than the amount of video data; therefore, multiple data packets of audio data are transmitted, followed by data packets of video data of equal size.

[0048] Transmitting audio and video data through a single channel avoids the need for separate channels, which would consume excessive bandwidth and ports, thus reducing performance overhead at both ends.

[0049] S102. Acquire the audio data and the video data, and play them synchronously.

[0050] Video data with added control icons can continue to be stored in the cache. When playback is needed, the audio and video data are retrieved from the cache and played synchronously. Alternatively, when playback is needed, the audio and video data are retrieved from the cache and played synchronously. Playback conditions can be set, such as starting playback when the received duration is greater than or equal to a preset duration threshold, or starting playback when the cached data volume is greater than or equal to a preset data volume threshold; where the received duration greater than or equal to the preset duration threshold and the cached data volume greater than or equal to the preset data volume threshold can be equivalent conditions. Synchronized playback of audio and video data allows for the display of a synchronized audio-visual video on the current electronic device's screen.

[0051] S103. Add a control icon to the playing video image. The control icon is used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0052] A control icon can be an identifying image used to control the playback of audio and video data. Typically, control icons are images of controls or icons. Video data is actually the media data being played and does not include interactive content. Adding a control icon to video data allows it to be displayed within the video image played on the current electronic device, thereby enabling the reception and response to user-input control commands based on that control icon.

[0053] In practice, adding control icons to a playing video can mean either rendering the control icons within the playing video frame, or rendering and displaying the control icons only when the user triggers an event to display them, and not rendering them when no such event is detected. The event that triggers the display of the control icons could be a user clicking or touching the screen of the current electronic device. Through human-computer interaction using the screen of the current electronic device, users can control the playback of synchronized audio and video in the video.

[0054] During playback, if a control icon is displayed on the screen of the current electronic device, the user can trigger the area where the control icon is located. The current electronic device receives the playback control command input by the user and controls the audio and video playback according to the playback control command, instead of waiting for changes in the received audio and video data before it can control the audio and video playback. This enables real-time control of the screen projection playback process.

[0055] Existing mobile screen mirroring functions use buffers to ensure smoothness in order to achieve audio-visual synchronization. However, this method is difficult to guarantee low latency. For example, if the screen on the playback end is sped up, the video on the mobile phone has already been sped up, but the playback end is still playing data in the buffer, so it cannot update the sped-up video content in real time.

[0056] According to the technical solution disclosed herein, by caching audio and video data and playing them synchronously when needed, the audio-video synchronization rate and playback quality can be improved. At the same time, control icons are added to the video data to receive playback control commands during playback, enabling real-time playback control on the projection end and reducing playback control latency.

[0057] Figure 2 This is a flowchart of another audio / video playback method disclosed in this disclosure, which is further optimized and extended based on the above technical solution and can be combined with the various optional implementation methods described above. The optimized audio / video playback method comprises: receiving a playback control command input by a user to the control icon; sending the playback control command to a data source terminal to cause the data source terminal to execute the playback control command, wherein the data source terminal is used to send the audio data and the video data; and, upon receiving a successful execution notification from the data source terminal, controlling the reception and buffering of data, and controlling the display of corresponding content on the screen according to the playback control command.

[0058] S201, Receive and buffer audio and video data.

[0059] S202. Acquire the audio data and the video data, and play them synchronously.

[0060] S203. Add a control icon to the playing video image. The control icon is used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0061] Add control icons to the playing video image, and render and display the control icons when a user clicks or touches the display screen of the current electronic device.

[0062] Optionally, acquiring the audio data and the video data and playing them synchronously includes: acquiring the audio data and the video data and playing them synchronously when the amount of cached data is greater than or equal to a preset data amount threshold.

[0063] After caching audio and video data, the system checks whether the cached data meets playback conditions. If the conditions are met, it begins acquiring audio and video data for synchronized playback. The playback condition is that the cached data volume is greater than or equal to a preset data volume threshold. For example, the data volume threshold can be expressed directly as the data size or as the duration. For instance, the data volume threshold could be 100MB. Or, the data volume threshold could be the amount of data in a 10-second video stream. The data volume threshold is related to the device type (e.g., model) and transmission bandwidth of the in-vehicle terminal. For example, the data volume threshold can be calculated based on the device type and transmission bandwidth of the in-vehicle terminal. This method of calculating the data volume threshold avoids the minimum latency of transmission via Wi-Fi and the corresponding audio / video (or pure video) data volume.

[0064] By configuring playback conditions, the vehicle terminal only starts to synchronously play the cached audio and video data after the data source has played for a certain duration. This avoids the transmission bandwidth affecting the smoothness of the audio and video playback on the vehicle terminal, ensuring the smoothness of audio and video playback on the vehicle terminal and improving playback quality.

[0065] S204, Receive playback control commands input by the user for the control icon.

[0066] Users can input playback control commands on the display screen. Users can input playback control commands directly on the screen, or they can input them after the control icons are rendered and displayed in the video image. Playback control commands can be trigger operations on a specific control icon, such as clicking, touching, pressing, or long-pressing. Alternatively, playback control commands can be control operations input by the user on the display screen, which may be unrelated to the control icon. For example, a control operation unrelated to the control icon could be a long press on any location on the display screen other than the control icon, or a swipe gesture outside the control icon.

[0067] For example, the playback control command can be determined based on the type of control icon triggered by the user. For example, if the control icon is a pause control image, the playback control command is to pause playback. If the control icon is a speed-up playback control image, the playback control command is to speed up playback.

[0068] S205, the playback control command is sent to the data source end so that the data source end executes the playback control command, and the data source end is used to send the audio data and the video data.

[0069] The data source projects its content onto the current electronic device for playback. After establishing a connection for screen projection control between the data source and the current electronic device, it begins playing audio and video data, transmitting the currently playing audio and video data to the current electronic device respectively. The data source is the device that actually controls the audio and video data, while the current electronic device is the projection device for the data source. The current electronic device dynamically changes in accordance with the content displayed on the data source. The current electronic device does not immediately play the received audio and video data; instead, it stores them in a buffer. Due to the buffering, the playback time of the current electronic device lags behind the playback time of the data source. The playback control commands control the playback process on the data source, and the playback process on the current electronic device changes accordingly.

[0070] Playback control commands are actually used to control the playback process at the data source end, and indirectly control the playback process of the current electronic device.

[0071] S206, upon receiving a successful execution notification from the data source, the system controls the reception and buffering of data, and controls the display of corresponding content on the screen according to the playback control instructions.

[0072] Successful execution indicates that the playback control command was executed successfully at the data source. Accordingly, the current electronic device should display the content indicating successful execution of the playback control command. Conversely, if execution fails, the current electronic device should display the content indicating failure to execute the playback control command.

[0073] Furthermore, the playback process at the data source end will affect the playback process of the current electronic device, which is actually determined by the received audio and video data. Therefore, the playback control commands are adapted by changing the current electronic device's reception of audio and video data.

[0074] For example, playback control commands may include: pause playback commands, resume playback commands, fast forward playback commands, or rewind playback commands, etc.

[0075] Optionally, the playback control instruction includes: a pause playback instruction; the step of controlling the reception and caching of data, and controlling the display of corresponding content on the screen according to the playback control instruction upon receiving a successful execution notification from the data source, includes: upon receiving a successful execution notification from the data source, stopping the playback of audio and video data, and stopping the reception and caching of new audio and video data; maintaining the display of the current video image on the screen, and displaying the pause playback status content on the current video image.

[0076] The Pause playback command is used to pause audio and video playback on the screen of the current electronic device. It stops receiving and buffering new audio and video data sent from the data source. Buffered data remains unchanged. The Pause playback command keeps the current video frame displayed; this refers to the frame of video that is paused.

[0077] The paused playback status message is used to inform the user that the audio / video is currently paused. This message can be a pause icon, such as an icon with two vertical lines. However, it can also be other content indicating a paused state; there are no specific limitations.

[0078] Optionally, the playback control instruction includes: a resume playback instruction; the step of controlling the reception and caching of data, and controlling the display of corresponding content on the screen according to the playback control instruction upon receiving a successful execution notification from the data source, includes: resuming playback of audio and video data upon receiving a successful execution notification from the data source, and resuming the reception and caching of new audio and video data; continuing to play video images on the screen, and stopping the display of content in the paused playback state.

[0079] The resume playback command is used to control the resumption of audio and video playback on the current electronic device's screen. The resume playback command is a command received after successfully executing a paused playback state. That is, the current electronic device is in a paused playback state, and the user wants the device to resume normal playback. The current electronic device continues to receive and buffer new audio and video data sent from the data source. The buffered data includes the received new audio and video data. The next frame of video begins playing on the screen, and the corresponding audio data plays synchronously. The current electronic device stops displaying the content from the paused playback state.

[0080] Optionally, the playback control instruction includes a fast-forward playback instruction; the step of controlling the reception and caching of data, and controlling the display of corresponding content on the screen according to the playback control instruction when the execution success signal is received from the data source includes: comparing the fast-forward duration corresponding to the fast-forward playback instruction with the caching duration corresponding to the data volume threshold when the execution success signal is received from the data source; controlling the reception of audio and video data corresponding to the fast-forward playback instruction sent by the data source, and processing the caching data according to the comparison result; and starting to play the audio and video data corresponding to the fast-forward playback instruction on the screen.

[0081] The total amount of cached audio and video data is less than or equal to the data size threshold. The cache duration is the playable duration of the audio and video corresponding to the data size threshold. The fast-forward playback command is used to play audio and video at a point in time after a certain duration has elapsed. It is typically used when the user skips a period of audio or video and plays the subsequent audio or video. The fast-forward duration refers to the time interval between the fast-forwarded point in time and the current playback point.

[0082] In practice, if the fast-forward duration is greater than or equal to the buffer duration, it indicates that the buffered data does not contain playable audio and video data corresponding to the fast-forward playback command. In this case, all buffered data is invalid, and the data source needs to send audio and video data starting from the fast-forward time point and continuing afterward. The corresponding control for receiving the audio and video data corresponding to the fast-forward playback command from the data source, and processing the buffered data, can be done by clearing the buffer and receiving new buffered data. If the fast-forward duration is less than the buffer duration, it indicates that the buffered data contains playable audio and video data corresponding to the fast-forward playback command. The corresponding control for receiving the audio and video data corresponding to the fast-forward playback command from the data source, and processing the buffered data, can be done by not processing the buffer and directly receiving new buffered data. In this case, synchronous playback of the corresponding time point can begin directly, and it is optional to delete the audio and video data in the buffer before the time point corresponding to the fast-forward playback command.

[0083] Optionally, if the fast-forward duration is greater than or equal to the buffer duration, the buffered audio and video data is cleared, and the audio and video data at the time point corresponding to the fast-forward playback command sent by the data source is received. If the amount of buffered data is greater than or equal to a preset data amount threshold, the buffered audio and video data is started to be played synchronously. If the fast-forward duration is less than the buffer duration, the audio and video at the time point corresponding to the fast-forward playback command is obtained and played synchronously.

[0084] Alternatively, when the fast-forward duration is less than the buffer duration, the corresponding control to receive the audio and video data corresponding to the fast-forward playback command sent by the data source, as well as to process the buffer data, can also be: control to clear the buffer, directly receive new buffer data, specifically clear the buffered audio and video data, and receive the audio and video data at the time point corresponding to the fast-forward playback command sent by the data source, and start synchronously playing the buffered audio and video data when the amount of buffered data is greater than or equal to a preset data amount threshold.

[0085] In addition, receiving a success message from the data source indicates that the current electronic device has notified the data source to send the audio and video data at the time point corresponding to the fast-forward playback command; or receiving a success message from the data source may mean that the current electronic device has not notified the data source to send the audio and video data at the time point corresponding to the fast-forward playback command, but after clearing the cache, the current electronic device additionally notifies the data source to send the audio and video data at the time point corresponding to the fast-forward playback command.

[0086] Optionally, the playback control command includes: a rewind playback command; the step of controlling the reception and caching of data, and controlling the display of corresponding content on the screen according to the playback control command when receiving a successful execution notification from the data source end includes: clearing the cache when receiving a successful execution notification from the data source end; notifying the data source end to send the data corresponding to the rewind playback command, and receiving and caching new audio data and new video data; and starting to play the audio data and video data corresponding to the rewind playback command on the screen.

[0087] The rewind command is used to play audio and video at a point in time after the previous playback command has been executed. This is suitable for users who want to rewind a certain amount of time to return to a previous point in time and play the corresponding audio and video. In practice, since the previously played audio and video has already been deleted (i.e., the corresponding storage resources have been released), the audio and video data for rewind playback is not in the cache. Therefore, the cached audio and video data can be directly deleted, and the data source needs to send the audio and video data from the point in time after the previous playback command. The cached audio and video data is cleared, and the data source receives the audio and video data at the time corresponding to the rewind command. Once the amount of cached data is greater than or equal to a preset data size threshold, synchronous playback of the cached audio and video data begins.

[0088] Furthermore, receiving a success message from the data source indicates that the current electronic device has notified the data source to send the audio and video data at the time point corresponding to the rewind playback command; or receiving a success message from the data source could mean that the current electronic device has not notified the data source to send the audio and video data at the time point corresponding to the rewind playback command, but after clearing the cache, the current electronic device additionally notifies the data source to send the audio and video data at the time point corresponding to the rewind playback command.

[0089] By assigning content to the current electronic device to execute under common control commands such as pause, resume, fast forward, and rewind, and by processing cached data and playback content accordingly, the system can flexibly handle various playback scenarios, provide zero-latency response to interactive control, and improve the accuracy and flexibility of playback control.

[0090] Other scenarios are also possible, such as speed-up or slow-down playback commands, without specific limitations. If a speed-up playback command from the data source is successfully executed, the cache can be cleared. The data source then processes the audio and video at the current playback time point on the electronic device at a faster speed and sends it to the current electronic device. When the amount of cached data is greater than or equal to a preset data volume threshold, the current electronic device begins synchronous playback of the cached audio and video data. Similarly, if a slow-down playback command from the data source is successfully executed, the cache can be cleared. The data source then processes the audio and video at the current playback time point on the electronic device at a slower speed and sends it to the current electronic device. When the amount of cached data is greater than or equal to a preset data volume threshold, the current electronic device begins synchronous playback of the cached audio and video data. Alternatively, the current electronic device can directly process the cached audio and video data at a faster or slower speed and receive the new audio and video data processed by the data source at a faster or slower speed.

[0091] Optionally, the synchronized playback includes: synchronously playing audio and video data on the vehicle's infotainment screen.

[0092] In a specific application scenario, the current electronic device is an in-vehicle device. Video data from the current electronic device is played on the vehicle's infotainment screen, along with synchronized audio data. The current electronic device serves as a projection terminal, used to play content from the data source.

[0093] By setting the application scenario to in-vehicle screen projection, synchronized audio and video playback can be achieved, while improving the smoothness of audio and video playback and reducing the control latency of in-vehicle screen projection.

[0094] According to the technical solution disclosed herein, the received playback control command is sent to the data source end through the current electronic device so that the data source end can execute it. When the data source end reports successful execution, the playback control command is executed, thus avoiding the execution of the command reflected in the audio and video data sent by the data source end, improving the real-time control performance and reducing the screen projection control delay.

[0095] Figure 3 This is a flowchart of another audio / video playback method disclosed in this embodiment, which is further optimized and extended based on the above technical solution and can be combined with the various optional implementation methods described above. Adding a control icon to the played video image specifically involves: acquiring a video image of the video data; acquiring the position of a control area and the corresponding control icon in the video image of the video data; and adding the control icon corresponding to the control area at the position of the control area on the played video image.

[0096] S301: Receive and buffer audio and video data.

[0097] S302. Acquire the audio data and the video data, and play them synchronously.

[0098] S303. Obtain the video image of the video data.

[0099] Video data can include multiple frames. Video data is transmitted in playback order; typically, the first frame received is the one from the beginning of the playback sequence, and playback begins synchronously from that first frame. The first frame, whether played or received, can be captured and identified as part of the video data.

[0100] S304. Obtain the position of the control area in the video image of the video data and the control icon corresponding to the control area.

[0101] In practice, when a video is playing at the data source, the data source adds a control icon to the video. This icon is then displayed on the screen when the user triggers the display, allowing the human-computer interaction system to receive user input and control the video playback. The data source sends the playing video and audio data to the current electronic device. The video data received by the current electronic device actually includes the control icon displayed at the data source. However, the control icon is only displayed when triggered by the user. The current electronic device intercepts this trigger event, preventing it from being transmitted to the data source. Consequently, the data source cannot respond to the trigger event and therefore cannot display the control icon. Consequently, the video data transmitted by the data source does not contain the control icon, and therefore the video image displayed on the current electronic device does not show the control icon added by the data source.

[0102] In a video image, the control region can refer to the image area containing the control icons added to the video from the data source. The control icons corresponding to the control region can refer to the control icons added to the video by the current electronic device.

[0103] S305. Add a control icon corresponding to the control area at the position of the control area on the video image being played. The control icon is used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0104] Typically, the video control functions of the current electronic device and the video control functions of the data source are the same or similar. For the same control area, the control icons added by the current electronic device and the data source have the same control functions. The control icon corresponding to the control area can refer to the control icon whose control function is corresponding to the control icon added by the data source in that control area. The control icon corresponding to the control area is added by the current electronic device in the control area. Accordingly, based on the recognition results of the control icons in the control area, the current electronic device can add control icons with the same function as the recognition results in the control area.

[0105] Among them, the control icons added to the current electronic device and the data source in the same control area can be different or the same, but the control functions are the same.

[0106] Optionally, the step of acquiring the position of the control region and the corresponding control icon in the video image of the video data includes: uploading the video image of the video data to the cloud; and receiving the position of the control region and the corresponding control icon from the cloud.

[0107] The detection of the control area can be achieved by the current electronic device or by sending it to the cloud, where a cloud server will handle the detection. The cloud receives the video image and identifies the control area within it, along with the control functions of the control icons added by the data source. Based on the identified control functions, the cloud generates the corresponding control icons.

[0108] By enabling control area recognition and control icon generation through the cloud, the detection operations of current electronic devices are reduced, the accuracy and efficiency of control area detection are improved, the processing speed of video data of current electronic devices is accelerated, and the playback response speed of electronic devices is improved.

[0109] According to the technical solution disclosed herein, by adding control icons to the video image being played, human-computer interaction control of audio and video playback can be achieved, thereby enabling low-latency control of audio and video data by current electronic devices.

[0110] Figure 4 This is a flowchart of an audio / video playback method disclosed in an embodiment of this disclosure. This embodiment can be applied to situations where the data source is projecting onto the controlled end. The method of this embodiment can be executed by an audio / video playback device, which can be implemented in software and / or hardware and specifically configured in an electronic device with a certain data processing capability. The electronic device can be a client device or a server device. Client devices include mobile phones, tablets, vehicle terminals, and desktop computers.

[0111] S401, Acquire video and audio data.

[0112] The data source encodes the audio and video data of the same video using H.264 and encapsulates the encoded video and audio data into RTMP protocol data packets according to the RTMP protocol encapsulation format. A communication channel is established between the data source and the controlled device, and the audio and video data are transmitted solely through this single transmission channel. The controlled device is the current electronic device described in the aforementioned embodiment.

[0113] S402. Send the video data and the audio data to the controlled terminal so that the controlled terminal receives and caches the audio data and video data; acquire the audio data and the video data and play them synchronously; add a control icon to the playing video image, the control icon being used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0114] Optionally, sending the video data and the audio data to the controlled terminal includes: establishing a communication channel with the controlled terminal; and sending the video data and the audio data to the controlled terminal through the communication channel.

[0115] There is only one communication channel. This channel is used to transmit both audio and video data. Transmitting audio and video data through a single channel avoids the need for separate channels, which would consume excessive bandwidth and ports, thus reducing performance overhead at both ends.

[0116] According to the technical solution disclosed herein, by caching audio and video data and playing them synchronously when needed, the audio-video synchronization rate and playback quality can be improved. At the same time, control icons are added to the video data to receive playback control commands during playback, enabling real-time playback control on the projection end and reducing playback control latency.

[0117] Figures 5-11 This is a scenario diagram illustrating an audio / video playback method provided by the technical solution of this disclosure. Specifically, the audio / video playback method is applied to projecting media data from a mobile phone to an in-vehicle terminal for playback.

[0118] (a) such as Figure 5 and Figure 6 The diagram illustrates RTMP encapsulation and transmission between an in-vehicle terminal and a mobile phone: A wireless connection is established between the mobile phone and the in-vehicle terminal, and a communication channel is set up to transmit data. The mobile phone then retrieves the audio and video source data of the same video file. Figure 6 The raw audio data PCM (Pulse Code Modulation) and the raw video data YUV ("Y" represents luminance, i.e., grayscale value Luminance or Luma, and "U" and "V" represent chrominance or Chroma) are compressed using h264 encoding to obtain encoded data, specifically including audio encoded data ( Figure 6 (AAC) and (Video) encoded data Figure 6The video data is encoded in H.264. The mobile device encapsulates the encoded data into RTMP-formatted messages according to the RTMP protocol's encapsulation format and segments them into data packets (chunks). Specifically, audio encoded data is encapsulated into audio messages and divided into audio data packets, and video encoded data is encapsulated into video messages and divided into video data packets. The mobile device only needs to transmit the audio and video data packets to the vehicle terminal through a single transmission channel and according to the transmission protocol.

[0119] In this process, audio data packets and video data are sent separately. Specifically, based on the RTMP protocol, the order in which data packets are sent in the MainSocket can be as follows: send multiple audio data packets, then send at least one video data packet of the same size, continue sending audio data packets, then send at least one video data packet of the same size, and so on.

[0120] (ii) Figure 7 and Figure 8 This demonstrates how the vehicle-mounted terminal receives and processes data:

[0121] A buffer pool (based on local file buffering) is configured in the vehicle terminal. This buffer pool caches received data packets; specifically, upon receiving a data packet, it is assembled into a message and then cached. The buffer pool's capacity is a data volume threshold. Once the buffer pool is full, indicating that the cached data volume is greater than or equal to the data volume threshold, the vehicle terminal's player is notified to start reading data from the buffer pool and begin playback. The data volume threshold is 100MB, which can be set as needed, or it can be the data volume of a 15-second video. If the buffer pool is not full, indicating that the cached data volume is less than the data volume threshold, no action is taken, and caching continues.

[0122] In the vehicle terminal, the player retrieves messages from the buffer pool, obtains audio and video encoded data, and decodes them to obtain source data composed of raw audio and video data. The player then begins to play the audio and video source data synchronously.

[0123] Check if image recognition has been performed on the video. If so, it means the image has passed. Figure 9 and Figure 10The system obtains the coordinates of the control icon (key icon) and the control area. OpenGL can then be used to add the control icon to the control area of ​​the raw video data. If image recognition is not performed, the location of the control area is identified, the corresponding control icon is obtained, and OpenGL can be used to add the control icon to the control area of ​​the raw video data. The added raw video and audio data are sent to the player. The player, based on the timestamps carried in the RTMP protocol and format, performs synchronized audio and video playback. It should be noted that adding the control icon to the raw video data is not the same as rendering the control icon directly on the video image. You can choose to render and display the control icon directly on all video images in the raw video data, or you can render and display the control icon only on the currently displayed video image when a trigger event is detected.

[0124] (III) Figure 9 and Figure 10 This demonstrates cloud-based OCR (Optical Character Recognition) image recognition:

[0125] When the in-vehicle terminal player starts playing the first frame of video image, OCR image recognition is required to transmit the first frame of video data to the cloud.

[0126] The cloud uses OCR image recognition technology to save key icons (such as rewind, play, and pause) from the first frame of video image, finds similar icons, i.e. control icons, and obtains the position coordinates of the key icons in the first frame of video image, i.e., determines the control area.

[0127] The cloud sends the location coordinates of similar icons and control areas to the vehicle terminal, which then saves the location coordinates of the control areas and their corresponding control icons. There can be at least one control area, and each control area has a corresponding control icon.

[0128] like Figure 10 The video image shown is after adding control icons, which include control icons for functions such as exiting the video, switching to the next episode, and pausing.

[0129] (iv) such as Figure 11 The vehicle-mounted terminal and the mobile terminal achieve zero-latency interaction of control commands:

[0130] When a user taps the screen on the in-vehicle terminal, triggering an event that displays control icons, the terminal draws the icons onto the currently playing video and displays the control icons (such as rewind, play, or pause) on the video. Figure 10 As shown.

[0131] If the user then clicks the pause icon again, the in-vehicle terminal will determine whether the user has clicked the pause control icon based on the click location. If so, the in-vehicle terminal will notify the mobile phone to pause video playback (the video will pause). The mobile phone can then control the pause and playback of audio and video.

[0132] When the in-vehicle terminal receives a successful pause command from the mobile phone, it will no longer store the data packets sent by the mobile phone in the buffer pool. Simultaneously, the in-vehicle terminal will not read data from the buffer pool, and will draw a marker image indicating the paused state on the currently playing video image to notify the user of successful pause. In essence, pausing playback means that the in-vehicle terminal no longer reads data from the buffer pool, nor caches data in the buffer pool, maintaining a static state.

[0133] After pausing, if the user clicks the pause icon or another play icon, the in-vehicle terminal will determine whether the user clicked the play control icon based on the click location. If so, the in-vehicle terminal will notify the mobile phone to resume video playback (the video will resume playback). The mobile phone will then control the resumption of audio and video playback. The in-vehicle terminal will stop drawing the icon indicating the paused state. Upon receiving the command from the mobile phone that playback has been successfully resumed, the in-vehicle terminal will store the data packet sent by the mobile phone in a buffer pool; simultaneously, the in-vehicle terminal will read data from the buffer pool.

[0134] For example, swiping right on the in-vehicle terminal screen (the fast-forward swipe gesture) determines the fast-forward duration based on the swipe distance and duration. The fast-forward duration is compared to the buffer pool's buffering time, for example, 15 seconds. If fast-forwarding exceeds 15 seconds (exceeding the buffer pool's buffering time limit), the buffer pool is immediately cleared, and the mobile device is notified to play and send the audio and video 15 seconds ahead. If the buffer pool is full (i.e., the cached data amount is greater than or equal to the data amount threshold, meeting the playback condition), the player is notified to immediately begin synchronously playing the newly cached audio and video, i.e., fast-forwarding to a 15-second segment of video.

[0135] For example, when swiping left on the in-vehicle terminal screen (the back swipe gesture), the back swipe duration can be determined based on the swipe distance and duration. The buffer pool is immediately cleared, and the mobile device is notified to play and send the audio and video 15 seconds later. If the buffer pool is full (i.e., the cached data volume is greater than or equal to the data volume threshold, meeting the playback condition), the player is notified to immediately begin synchronously playing the newly cached audio and video, i.e., the video segment that rewinds to 15 seconds.

[0136] According to the technical solution disclosed herein, only one core communication channel needs to be created to transmit the poem, eliminating the need to open separate channels for video and audio, thus reducing the performance overhead of both ends; audio and video synchronization can be achieved for video and multimedia playback scenarios, ensuring a good viewing experience; a buffer function is implemented in the vehicle terminal to ensure smooth playback; and OCR image recognition technology is implemented in the vehicle terminal to record the key icons and coordinates of the video itself, and then draw the key icons through OpenGL to simulate the real page of a mobile phone, achieving zero-latency control interaction.

[0137] According to embodiments of this disclosure, Figure 12 This is a structural diagram of the audio and video playback device according to an embodiment of this disclosure. This embodiment is applicable to the synchronous playback of audio and video data of the same video. The device is implemented in software and / or hardware and is specifically configured in an electronic device with a certain data processing capability.

[0138] like Figure 12 The audio / video playback device 1200 shown includes: a data receiving module 1201, a synchronous playback module 1202, and a control icon adding module 1203. Among them,

[0139] Data receiving module 1201 is used to receive and buffer audio and video data;

[0140] The synchronous playback module 1202 is used to acquire the audio data and the video data and perform synchronous playback;

[0141] The control icon adding module 1203 is used to add control icons to the playing video image. The control icons are used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0142] According to the technical solution disclosed herein, by caching audio and video data and playing them synchronously when needed, the audio-video synchronization rate and playback quality can be improved. At the same time, control icons are added to the video data to receive playback control commands during playback, enabling real-time playback control on the projection end and reducing playback control latency.

[0143] Furthermore, the audio and video playback device further includes: a control instruction receiving module, used to receive playback control instructions input by the user for the control icon; a control instruction sending module, used to send the playback control instructions to the data source end, so that the data source end executes the playback control instructions, the data source end being used to send the audio data and the video data; and a control instruction execution module, used to, upon receiving a successful execution notification from the data source end, control the receiving and buffering of data, and control the display of corresponding content on the screen according to the playback control instructions.

[0144] Furthermore, the synchronous playback module 1202 includes a buffer quantity detection unit, used to acquire the audio data and the video data and perform synchronous playback when the amount of buffered data is greater than or equal to a preset data quantity threshold.

[0145] Furthermore, the playback control command includes a pause playback command; the control command execution module includes a stop caching unit, used to stop playing the audio data and the video data, and to stop receiving and caching new audio data and new video data, upon receiving a successful execution notification from the data source; and a stop playback unit, used to keep the current video image displayed on the screen and to display the pause playback status on the current video image.

[0146] Furthermore, the playback control command includes: a resume playback command; the control command execution module includes: a resume cache unit, used to resume playback of the audio data and the video data upon receiving a successful execution notification from the data source, and to resume receiving and cache new audio data and new video data; and a resume playback unit, used to continue playing the video image on the screen and stop displaying content in the paused playback state.

[0147] Furthermore, the playback control command includes a fast-forward playback command; the control command execution module includes a fast-forward duration comparison unit, used to compare the fast-forward duration corresponding to the fast-forward playback command with the buffer duration corresponding to the data volume threshold when the execution is successfully received from the data source; a fast-forward control unit, used to control the receiving of audio and video data corresponding to the fast-forward playback command from the data source and to process the buffer data according to the comparison result; and a fast-forward playback unit, used to start playing the audio and video data corresponding to the fast-forward playback command on the screen.

[0148] Furthermore, the playback control command includes a rewind playback command; the control command execution module includes a cache clearing unit, used to clear the cache upon receiving a successful execution notification from the data source; a rewind control unit, used to receive and cache the audio and video data corresponding to the rewind playback command sent by the data source; and a rewind playback unit, used to start playing the audio and video data corresponding to the rewind playback command on the screen.

[0149] Furthermore, the control icon adding module 1203 includes: an image acquisition unit for acquiring a video image of the video data; a control icon acquisition unit for acquiring the position of a control area and the control icon corresponding to the control area in the video image of the video data; and a control icon acquisition unit for adding the control icon corresponding to the control area at the position of the control area on the playing video image.

[0150] Furthermore, the control icon acquisition unit includes: an image uploading subunit for uploading video images of the video data to the cloud; and a cloud recognition subunit for receiving the location of the control area and the control icon corresponding to the control area from the cloud.

[0151] Furthermore, the data receiving module 1201 includes a single-channel data transmission unit for acquiring audio and video data sent by the data source end on the same communication channel.

[0152] Furthermore, the synchronous playback module 1202 includes: a vehicle-mounted synchronous playback unit, used to synchronously play audio and video data on the vehicle-mounted screen.

[0153] The above-described audio and video playback device can execute the audio and video playback method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the audio and video playback method.

[0154] According to embodiments of this disclosure, Figure 13 This is a structural diagram of the audio / video playback device according to an embodiment of this disclosure. This embodiment is applicable to training audio / video playback models. The device is implemented using software and / or hardware and is specifically configured in an electronic device with certain data processing capabilities.

[0155] like Figure 13 The audio / video playback device 1300 shown includes: a source data acquisition module 1301 and a data transmission module 1302. Wherein,

[0156] The source data acquisition module 1301 is used to acquire video data and audio data;

[0157] The data transmission module 1302 is used to send the video data and the audio data to the controlled terminal, so that the controlled terminal receives and buffers the audio data and video data; acquires the audio data and the video data and plays them synchronously; and adds a control icon to the video image being played, the control icon being used to receive playback control instructions to control the audio and video playback according to the playback control instructions.

[0158] According to the technical solution disclosed herein, by caching audio and video data and playing them synchronously when needed, the audio-video synchronization rate and playback quality can be improved. At the same time, control icons are added to the video data to receive playback control commands during playback, enabling real-time playback control on the projection end and reducing playback control latency.

[0159] Furthermore, the data transmission module 1302 includes: a single-channel establishment unit for establishing a communication channel with the controlled terminal; and a single-channel transmission unit for sending the video data and the audio data to the controlled terminal through the communication channel.

[0160] The above-described audio and video playback method apparatus can execute the audio and video playback method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the audio and video playback method.

[0161] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0162] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0163] Figure 14 A schematic area diagram of an example electronic device 1400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0164] like Figure 14As shown, device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1402 or a computer program loaded from storage unit 1408 into random access memory (RAM) 1403. The RAM 1403 may also store various programs and data required for the operation of device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.

[0165] Multiple components in device 1400 are connected to I / O interface 1405, including: input unit 1406, such as a keyboard, mouse, etc.; output unit 1407, such as various types of displays, speakers, etc.; storage unit 1408, such as a disk, optical disk, etc.; and communication unit 1409, such as a network card, modem, wireless transceiver, etc. Communication unit 1409 allows device 1400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0166] The computing unit 1401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as audio and video playback methods. For example, in some embodiments, the audio and video playback method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the audio and video playback method described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to perform audio and video playback methods by any other suitable means (e.g., by means of firmware).

[0167] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard objects (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or area diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0169] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0171] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0172] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and weak business scalability inherent in traditional physical hosting and VPS services. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0173] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0174] Cloud computing refers to a technology system that enables access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0175] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0176] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An audio-video playing method, comprising: receiving and buffering audio data and video data; acquiring the audio data and the video data and playing them synchronously; acquiring a video image of the video data; wherein no control icon added by a data source end is displayed in the video image; uploading the video image of the video data to a cloud end; wherein the cloud end identifies a control region in the video image and a control function of a control icon in the control region added by the data source end based on the received video image, and generates a control icon corresponding to the control function based on the control function; receiving a position of the control region and a control icon corresponding to the control region fed back by the cloud end; adding the control icon corresponding to the control region at the position of the control region on the played video image; receiving a playing control instruction input by a user for the control icon; wherein the playing control instruction is used to control a playing process of the data source end; sending the playing control instruction to the data source end, so that the data source end executes the playing control instruction, and the data source end is used to send the audio data and the video data; in a case where an execution result of the playing control instruction sent by the data source end is execution success, controlling receiving and buffering the audio data and the video data according to the playing control instruction to control audio-video playing and controlling displaying corresponding content on a screen.

2. The method of claim 1, wherein, the acquiring the audio data and the video data and playing them synchronously comprises: when a buffered data amount is greater than or equal to a preset data amount threshold, acquiring the audio data and the video data and playing them synchronously.

3. The method of claim 1, wherein, the playing control instruction comprises a pause playing instruction; in the case where the execution result of the playing control instruction sent by the data source end is execution success, the controlling receiving and buffering data and the controlling displaying corresponding content on the screen according to the playing control instruction comprises: in the case where the execution result of the playing control instruction sent by the data source end is execution success, stopping playing the audio data and the video data and stopping receiving and buffering new audio data and new video data; stopping displaying a current video image on the screen and displaying content in a pause playing state on the current video image.

4. The method of claim 1, wherein, the playing control instruction comprises a resume playing instruction; in the case where the execution result of the playing control instruction sent by the data source end is execution success, the controlling receiving and buffering data and the controlling displaying corresponding content on the screen according to the playing control instruction comprises: in the case where the execution result of the playing control instruction sent by the data source end is execution success, resuming playing the audio data and the video data and resuming receiving and buffering new audio data and new video data; continuing playing a video image on the screen and stopping displaying content in a pause playing state.

5. The method of claim 2, wherein, the playing control instruction comprises a fast-forward playing instruction; in the case where the execution result of the playing control instruction sent by the data source end is execution success, the controlling receiving and buffering data and the controlling displaying corresponding content on the screen according to the playing control instruction comprises: In the case of receiving the execution success sent by the data source end, the fast forward play instruction corresponding to the fast forward time length and the data volume threshold corresponding to the cache time length are compared; According to the comparison result, the audio data and video data corresponding to the fast forward play instruction sent by the data source end are received and cached, and the cached data is processed; Start playing the audio data and video data corresponding to the fast forward play instruction on the screen.

6. The method of claim 1, wherein, The play control instruction includes a backward play instruction. In the case of receiving the execution success sent by the data source end, according to the play control instruction, the data received and cached, and the corresponding content displayed on the screen, including: In the case of receiving the execution success sent by the data source end, clean up the cache; Receive and cache the audio data and video data corresponding to the backward play instruction sent by the data source end; Start playing the audio data and video data corresponding to the backward play instruction on the screen.

7. The method of claim 1, wherein, The received audio data and video data include: Acquire the audio data and video data sent by the data source end in the same communication channel.

8. The method of claim 1, wherein, The synchronous play includes: Synchronously play audio and video data on the car screen.

9. An audio and video play method, comprising: Acquire video data and audio data; Send the video data and the audio data to the controlled end, so that the controlled end receives and caches audio data and video data; Acquire the audio data and the video data and synchronously play; add a control icon in the played video image; Receive the play control instruction input by the user for the control icon; send the play control instruction to the data source end; wherein the play control instruction is used to control the play process of the data source end; Receive the play control instruction and execute, and send the execution result to the controlled end, so that the controlled end receives the execution result sent by the data source end in the case of execution success, according to the play control instruction, controls receiving and caching the audio data and the video data, controls audio and video play, and controls displaying the corresponding content on the screen; Wherein, adding a control icon in the played video image includes: Acquire the video image of the video data; wherein the control icon added by the data source end is not displayed in the video image; Upload the video image of the video data to the cloud; wherein the cloud identifies the control area in the video image and the control function of the control icon added by the data source end in the control area based on the received video image, and generates the control icon corresponding to the control function based on the control function; Receive the position of the control area and the control icon corresponding to the control area fed back by the cloud; Add the control icon corresponding to the control area at the position of the control area in the played video image.

10. The method of claim 9, wherein, The video data and the audio data sent to the controlled end include: Establish a communication channel with the controlled end; Send the video data and the audio data to the controlled end through the communication channel.

11. An audio and video play device, comprising: The data receiving module is configured to receive and buffer audio data and video data. The synchronous playing module is configured to acquire the audio data and the video data and play them synchronously. The control icon adding module is configured to add a control icon in a played video image, the control icon being configured to receive a playing control instruction to control audio and video playing according to the playing control instruction. The control instruction receiving module is configured to receive a playing control instruction input by a user for the control icon, wherein the playing control instruction is configured to control a playing process of a data source end. The control instruction sending module is configured to send the playing control instruction to the data source end to enable the data source end to execute the playing control instruction, wherein the data source end is configured to send the audio data and the video data. The control instruction executing module is configured to, in a case where an execution result of the playing control instruction sent by the data source end is execution success, control receiving and buffering of the audio data and the video data and control displaying of corresponding content on a screen according to the playing control instruction. The control icon adding module includes: The image acquiring unit is configured to acquire a video image of the video data, wherein the video image does not display a control icon added by a data source end. The control icon acquiring unit is configured to acquire a position of a control region in the video image of the video data and a control icon corresponding to the control region. The control icon acquiring unit is configured to add the control icon corresponding to the control region at the position of the control region in the played video image, wherein the control icon acquiring unit includes: The image uploading subunit is configured to upload the video image of the video data to a cloud end, wherein the cloud end is configured to identify the control region in the video image and a control function of the control icon added by the data source end in the control region based on the received video image, and generate the control icon corresponding to the control function based on the control function. The cloud end identification subunit is configured to receive the position of the control region and the control icon corresponding to the control region fed back by the cloud end.

12. The apparatus of claim 11, wherein, The synchronous playing module includes: The buffer amount detecting unit is configured to acquire the audio data and the video data and play them synchronously in a case where a buffered data amount is greater than or equal to a preset data amount threshold.

13. The apparatus of claim 11, wherein, The playing control instruction includes a pause playing instruction. The control instruction executing module includes: The stop buffering unit is configured to, in a case where the execution success is received from the data source end, stop playing the audio data and the video data and stop receiving and buffering new audio data and new video data. The stop playing unit is configured to keep displaying a current video image on the screen and display content in a pause playing state on the current video image.

14. The apparatus of claim 11, wherein, The playing control instruction includes a resume playing instruction. The control instruction executing module includes: The resume buffering unit is configured to, in a case where the execution success is received from the data source end, resume playing the audio data and the video data and resume receiving and buffering new audio data and new video data. The resuming playing unit is configured to continue playing the video image on the screen and stop displaying the content in the pause playing state.

15. The apparatus of claim 12, wherein, The playing control instruction comprises a fast-forward playing instruction. The control instruction execution module comprises: The fast-forward time length comparison unit is configured to compare the fast-forward time length corresponding to the fast-forward playing instruction with the cache time length corresponding to the data amount threshold, in the case that the execution success sent by the data source end is received. The fast-forward control unit is configured to control the receiving and caching of the audio data and the video data corresponding to the fast-forward playing instruction sent by the data source end and the processing of the cached data according to the comparison result. The fast-forward playing unit is configured to start playing the audio data and the video data corresponding to the fast-forward playing instruction on the screen.

16. The apparatus of claim 11, wherein, The playing control instruction comprises a backward playing instruction. The control instruction execution module comprises: The cache cleaning unit is configured to clean the cache in the case that the execution success sent by the data source end is received. The backward control unit is configured to receive and cache the audio data and the video data corresponding to the backward playing instruction sent by the data source end. The backward playing unit is configured to start playing the audio data and the video data corresponding to the backward playing instruction on the screen.

17. The apparatus of claim 11, wherein, The data receiving module comprises: The single-channel data transmission unit is configured to acquire the audio data and the video data sent by the data source end in the same communication channel.

18. The apparatus of claim 11, wherein, The synchronous playing module comprises: The car machine synchronous playing unit is configured to synchronously play the audio and video data on the car machine screen.

19. An audio and video playing device, comprising: A source data acquisition module is configured to acquire video data and audio data. A data transmission module is configured to send the video data and the audio data to a controlled end, so that the controlled end receives and caches the audio data and the video data, acquires the audio data and the video data, and plays them synchronously, adds a control icon in the playing video image, receives a playing control instruction input by a user for the control icon, sends the playing control instruction to a data source end, and uses the playing control instruction to control the playing process of the data source end. An instruction execution module is configured to receive and execute the playing control instruction, and send an execution result to the controlled end, so that the controlled end controls the receiving and caching of the audio data and the video data according to the playing control instruction to control the audio and video playing and controls the display of corresponding content on the screen in the case that the execution result sent by the data source end is received and the execution result is execution success. Adding a control icon in the playing video image comprises: Acquiring a video image of the video data, wherein the video image does not display a control icon added by the data source end. Uploading the video image of the video data to a cloud end, wherein the cloud end identifies a control region in the video image and a control function of a control icon added by the data source end in the control region based on the received video image, and generates a control icon corresponding to the control function based on the control function. Receiving the position of the control region and the control icon corresponding to the control region fed back by the cloud end. Adding a control icon corresponding to the control region at a position of the control region on the played video image.

20. The apparatus of claim 19, wherein, The data transmission module comprises: A single-channel establishing unit, configured to establish a communication channel with a controlled terminal; A single-channel transmission unit, configured to send the video data and the audio data to the controlled terminal through the communication channel. 21.An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the audio-video playing method of any one of claims 1-10.

22. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the audio-video playing method of any one of claims 1-10. 23.A computer program product comprising a computer program which, when executed by a processor, implements the audio-video playing method of any one of claims 1-10.

Images