Screen projection method, system and equipment and storage medium

Abstract

The embodiment of the invention provides a screen projection method and device, equipment and a storage medium. The method comprises the steps of determining an application category to which a target application belongs and a target frame rate corresponding to the application category; performing screen recording according to the target frame rate to obtain a target video; and sending the target video to a vehicle-mounted terminal, wherein the vehicle-mounted terminal is used for playing the target video. By implementing the method, the mobile terminal can reasonably set the frame rate during screen recording of the mobile terminal according to the category of the application displayed on the interface during screen projection, so that the amount of data needing to be transmitted to the vehicle-mounted terminal by the mobile terminal is reduced, and the consumption of electric energy of the mobile terminal can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of Internet of Vehicles, and in particular to a screen projection method, system, device and storage medium. Background Art

[0002] While driving, users often connect mobile terminals such as mobile phones to vehicle-mounted terminals through Bluetooth or WIFI hotspots, and project the screen of the mobile terminal onto the display screen of the vehicle-mounted terminal to reduce distraction and improve driving safety.

[0003] However, during the screen projection process, the mobile terminal needs to transmit a large amount of data to the vehicle-mounted terminal, which will consume a large amount of power of the mobile terminal. Summary of the invention

[0004] One purpose of the present application is to provide a screen projection method, system, device and storage medium, which has the advantage that the frame rate of the mobile terminal during screen recording can be reasonably set according to the category of application displayed on the interface of the mobile terminal when the mobile terminal is projecting the screen, thereby reducing the amount of data that the mobile terminal needs to transmit to the vehicle-mounted terminal, and can effectively reduce the power consumption of the mobile terminal.

[0005] Another object of the present application is to provide a screen projection method, device, equipment and storage medium, which has the advantage that the frame rate of the mobile terminal during screen recording can be reasonably set according to the user's user behavior (such as the user's line of sight and whether the user operates the screen of the vehicle-mounted terminal), thereby more intelligently providing the user with a projection screen that can meet the user's real-time needs, saving system resources while providing the user with a better user experience.

[0006] To achieve the above-mentioned objectives, in the first aspect, the present application provides a screen projection method, which is applied to a mobile terminal, and the method includes: determining an application category to which a target application belongs and a target frame rate corresponding to the application category; the target application is an application corresponding to an application interface currently displayed on the mobile terminal, and the application category characterizes the functional characteristics of the target application; performing screen recording at the target frame rate to obtain a target video; and sending the target video to a vehicle-mounted terminal, which is used to play the target video.

[0007] This method determines the application category of the application displayed on the current interface of the mobile terminal during the screen projection process between the mobile terminal and the vehicle-mounted terminal, sets the screen recording frame rate of the mobile terminal in a targeted manner, and sends the video recorded at the screen recording frame rate to the vehicle-mounted terminal in real time, and displays it to the user through the screen interface of the vehicle-mounted terminal. This can reduce the amount of data transmitted from the mobile terminal to the vehicle-mounted terminal, and can effectively reduce the power consumption of the mobile terminal.

[0008] In this implementation, the application category to which the target application belongs and the target frame rate corresponding to the application category include:

[0009] ① The first category, the applications included in the first category are video playback applications; the target frame rate corresponding to the first category is the first frame rate value. For example, the video playback application can be a video software like "Youku" or "iQiyi", or a software with video playback function like "WeChat" or "Weibo", which is not limited in this application. It is understandable that in order to ensure that users have a good user experience when watching videos, the frame rate corresponding to video applications needs to reach 25 frames per second or more. Specifically, the first frame rate value can be 25, 26 or other similar frame rate values, which is not limited in this application.

[0010] ② The second category, the applications included in the second category are navigation applications; the target frame rate corresponding to the second category is the second frame rate value. The target frame rate corresponding to the second category is the second frame rate value. For example, the video playback application can be a third-party map software such as "AutoNavi Map" and "Baidu Map", or a web version of a navigation or map application, which is not limited in this application. It is understandable that when users use navigation applications, they do not pay as much attention to navigation applications as they do to video playback applications. Therefore, for navigation applications, a frame rate of 1 to 5 frames per second is sufficient to meet user needs in most cases. Therefore, the second frame rate value can be 4, 5 or other similar frame rate values, which is not limited in this application.

[0011] ③ Other categories, the applications included in the other categories are applications that are neither in the first category nor in the second category; the target frame rate corresponding to the other categories is a third frame rate value. It should be understood that, in addition to video playback applications and navigation applications, for other applications, such as music playback software or built-in call applications on mobile phones, etc., most of the time, users may not pay attention to the screen information of these applications. Therefore, for these applications, the frame rate of the video only needs 1 to 2 frames per second when projecting the screen to meet the needs of users. Specifically, the third frame rate can be 4, 5 or other similar frame rate values, which is not limited in this application.

[0012] In this way, the screen recording frame rate of the mobile terminal can be set in a targeted manner according to the specific category and function of the application, the amount of data transmitted from the mobile terminal to the vehicle-mounted terminal can be greatly reduced, and the power consumption of the mobile terminal will also be effectively reduced.

[0013] It is understandable that different users have different personal preferences. For different users, the types of applications in their mobile terminals are different, and different users have different usage habits for the same application; for example, some users may be accustomed to frequently checking navigation routes. For such users, a frame rate of 1 to 5 frames per second may not be sufficient to meet user needs. Therefore, in this embodiment, the user can classify the applications in the mobile terminal in advance and set the corresponding frame rate for each category. In this way, in the subsequent screen projection process, the mobile terminal will obtain the category and frame rate corresponding to the application based on the currently displayed application, and record the screen according to the frame rate, bringing a better user experience to the user.

[0014] During the actual screen projection process, the user may make calls, switch songs, switch applications, etc. At this time, the user often needs to touch the projection interface (that is, the display of the vehicle terminal) to complete these operations. At this time, in order to achieve smooth operation, it is necessary to ensure that the projection frame rate is large enough, for example, the frame rate needs to be greater than 15 frames per second. Therefore, in this embodiment, when the user touches the projection interface, the vehicle terminal will send a first message to the mobile terminal, and the first message indicates that the user is currently touching the projection interface. When the mobile terminal receives the first message, the mobile terminal determines the target frame rate as the fourth frame rate value.

[0015] It should be understood that during the actual driving of the vehicle, even if the mobile terminal has been projecting the screen to the vehicle-mounted terminal, the user may not always be looking at the screen. Therefore, in this embodiment, the vehicle-mounted terminal can use an eye tracking algorithm to determine whether the user is currently looking at the display of the vehicle-mounted terminal (i.e., the projection interface). When the user is not currently looking at the projection interface, the vehicle-mounted terminal will send a second message to the mobile terminal. When the mobile terminal receives the second message, the mobile terminal can temporarily interrupt the projection process. It should be noted that the mobile terminal can interrupt the projection process by stopping screen recording; in addition, the mobile terminal can also interrupt the projection process by maintaining the screen recording action, but stopping sending the recorded video to the vehicle-mounted terminal. This application does not limit this.

[0016] Correspondingly, in this embodiment, the mobile terminal can also determine whether it has received the third information sent by the vehicle-mounted terminal, and the third information indicates that the user is currently looking at the screen projection interface; when the third information is received, the screen projection process is resumed.

[0017] In the second aspect, an embodiment of the present application provides a screen projection system, including a mobile terminal and a vehicle-mounted terminal, the mobile terminal including a first display, and the vehicle-mounted terminal including a second display; the mobile terminal is used to determine the application category to which the target application belongs and the target frame rate corresponding to the application category, and is used to screen record the interface displayed on the first display at the target frame rate to obtain a target video, and send the target video to the vehicle-mounted terminal; the target application is the application corresponding to the application interface currently displayed by the mobile terminal, and the application category characterizes the functional characteristics of the target application; the vehicle-mounted terminal is used to receive the target video sent by the mobile terminal, and use the second display to play the target video.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program is executed, the processor is used to execute the method in the first aspect and any optional implementation of the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on one or more processors, the method in the first aspect and any optional implementation of the first aspect is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the background technology.

[0021] Figure 1A A schematic diagram of a screen projection system provided in an embodiment of the present application;

[0022] Figure 1B A schematic diagram of a screen projection process provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a screen projection method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a scenario of screen projection based on a screen projection system provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a screen projection device provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings.

[0028] The terms "first" and "second" in the specification, claims and drawings of this application are only used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0029] The "embodiment" mentioned in this article means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It can be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] In the present application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three and more than three, and "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c".

[0031] The embodiments of the present application provide a screen projection method, system, device and storage medium, and in order to more clearly describe the solution of the present invention, the following first introduces some knowledge involved in the screen projection method, system, device and storage medium provided in the embodiments of the present application.

[0032] (1) The principle of screen projection

[0033] Generally, two terminals that support screen projection can complete screen projection through mirroring mode. For example, after a mobile terminal establishes a communication connection with a terminal device that supports screen projection, such as a vehicle-mounted terminal, the mobile terminal can continuously take screenshots or record the screen, and then send the recorded video to the vehicle-mounted terminal. In this way, the screen of one terminal can be synchronously displayed on the screen of another terminal. This is the basic principle of screen projection.

[0034] (2) Eye tracking

[0035] The gaze point of the human eye is determined by two factors: the orientation of the head and the orientation of the eyes. The basic principle of eye tracking technology is to use image processing technology, use an eye camera that can lock the eyes, and continuously record the changes in sight by taking in infrared rays reflected from the human cornea and pupil, so as to achieve the purpose of recording and analyzing the sight tracking process; or first use a camera to obtain a human eye or face image, and then use software to locate and track the face and eyes in the image, so as to estimate the user's gaze position on the screen.

[0036] (3) Application category

[0037] In an embodiment of the present application, "application category" refers to a label added to an application based on the functional characteristics and usage scenarios of the application. There can be an application in each application category, and each application category has its own corresponding frame rate. The frame rate indicates that when the mobile terminal opens the application interface of the application and projects the application interface of the application to another terminal (such as a vehicle-mounted terminal), the screen recording frame rate is set for the mobile terminal based on the functional characteristics and usage scenarios of the application. This screen recording frame rate can be adapted to the functional characteristics and usage scenarios of the application, thereby reducing the amount of data transmission between the two terminals while ensuring user experience, thereby reducing the power consumption of the mobile terminal.

[0038] Mobile car networking products are lightweight products that are very convenient, fast and low-cost. However, a big pain point of this product is the power consumption of the mobile phone. Currently, mobile phone manufacturers have not implemented the function of continuing to project the screen after the mobile phone is turned off. In other words, the screen of the mobile phone needs to be on all the time during the process of projecting the screen to the car terminal. In addition, since the mobile phone realizes the screen projection function by connecting to the car terminal through Bluetooth and WIFI hotspot, the mobile phone consumes power very quickly during the process of transmitting data to the car terminal because the mobile phone needs to continuously transmit data and the amount of data is very large. For details, please refer to Figure 1A and Figure 1B .

[0039] Figure 1A A schematic diagram of a screen projection system provided in an embodiment of the present application. Figure 1AAs shown, the mobile terminal 101 can be an electronic device such as a mobile phone, a tablet, or a computer that supports wireless communication and screen projection functions; the vehicle terminal 102 can be an electronic device that has wireless communication and screen projection functions in ordinary vehicles, special vehicles, or rescue vehicles. The vehicle terminal 102 can be connected to the mobile terminal 101 through Bluetooth or WIFI for wireless communication. Among them, the mobile terminal 101 is installed with a variety of device-provided application software and other third-party application software, including: video playback application 1011, which can be used to play videos stored in the mobile phone or access and play video files on the Internet through the mobile network; navigation application 1012, which can be used for location search and route query functions, while providing real-time voice navigation, and can also display real-time itinerary route maps and real-time intersection details for users; music playback application 1013, which can be used to play various music files, and supports fast forward (previous song), fast rewind (next song), lyrics display, volume adjustment, playback background, display spectrum, search and other operations. The call application 1014 can be used to make remote calls with others and supports functions such as dialing, number location query, hanging up, and contact storage.

[0040] exist Figure 1A In the screen projection system shown, when the vehicle terminal 102 establishes a communication connection with the mobile terminal 101, the mobile terminal 101 will record the screen image and transmit the recorded video to the vehicle terminal via the wireless local area network. After that, ignoring the delay caused by information transmission, the screen information displayed on the screen of the vehicle terminal 102 will be synchronized with the screen information displayed on the screen of the mobile terminal 101.

[0041] by Figure 1B Take this as an example to illustrate. Figure 1B A schematic diagram of a screen projection process provided in an embodiment of the present application. Figure 1B As shown, Figure 1B The vehicle-mounted terminal 102 and the mobile terminal 101 are Figure 1A The vehicle-mounted terminal 102 and the mobile terminal 101 shown in FIG. 1 are connected in communication with each other. Figure 1BIn the figure, the user interface 10 currently displayed by the mobile terminal 101 is the application interface of the above-mentioned music player application 103. As shown in the user interface 10, at this time, the user is playing a song through the mobile terminal, and the user interface 10 can display the playback background, spectrum and other information of the song. In addition, there is a screen recording mark 10A in the user interface 10, which indicates that the mobile terminal 101 is currently recording the screen at a certain frame rate (25 frames per second). Correspondingly, after the mobile terminal 101 sends the recorded video to the vehicle terminal 102, the screen projection interface 11 will be displayed on the screen of the vehicle terminal. It can be seen that the same screen as the user interface 10 can be displayed in the screen projection interface 11. In addition, there is a screen projection mark 11A in the user interface 10, which indicates that the vehicle terminal 102 is currently communicating with other terminals (i.e., the mobile terminal 101), and the vehicle terminal 102 obtains the interface information of other terminals and displays it on its own screen.

[0042] Generally, when the mobile terminal 101 is recording the screen, no matter which application is opened in the mobile terminal, the mobile terminal 101 records the screen at a constant frame rate. Figure 1B In the example, when the application currently opened by the mobile terminal 101 is a music player application 1013, the mobile terminal will record the screen at a frame rate of 25 frames per second; if the application currently opened by the mobile terminal 101 is the above-mentioned navigation application 1012, the mobile terminal will still record the screen at a frame rate of 25 frames per second and send the recorded video to the vehicle terminal 102. It can be understood that since the screen of the mobile terminal 101 needs to remain on during the entire screen projection process, a large amount of power is consumed. When the mobile terminal 101 always maintains a large frame rate for screen recording, the mobile terminal 101 will also need to perform a large amount of video recording and video encoding, which will further consume a large amount of power of the mobile terminal 101; in addition, since the recorded video is transmitted to the vehicle terminal 102 via Bluetooth and WIFI hotspots, when the amount of data transmitted is too large, the mobile terminal 101 will naturally consume a large amount of power.

[0043] In view of the defects existing in the above-mentioned screen projection system and screen projection method, the embodiment of the present application provides a screen projection method, which can determine the application category of the application displayed on the current interface of the mobile terminal during the screen projection process between the mobile terminal and the vehicle terminal, set the screen recording frame rate of the mobile terminal in a targeted manner, and send the video recorded at the screen recording frame rate to the vehicle terminal in real time, and display it to the user through the screen interface of the vehicle terminal, which can reduce the amount of data transmitted from the mobile terminal to the vehicle terminal and effectively reduce the power consumption of the mobile terminal. For details, please refer to Figure 2 .

[0044] Figure 2A flowchart of a screen projection method provided in an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0045] 201. Determine an application category to which a target application belongs and a target frame rate corresponding to the application category.

[0046] The mobile terminal determines the application category to which the target application belongs and the target frame rate corresponding to the application category.

[0047] The mobile terminal may be a mobile phone, tablet computer, computer (such as a laptop computer, PDA, etc.), mobile Internet device, wireless terminal in unmanned driving, etc. with data receiving and sending functions and screen projection functions. In addition, the mobile terminal may also be a device in an Internet of Things system. In another possible implementation, the terminal device shown in this application may include an access point, etc. It is understandable that this application does not limit the specific form of the mobile terminal. Any device that can communicate with a roadside device, a vehicle, or a vehicle management platform, etc., falls within the protection scope of the terminal device.

[0048] The target application is the application corresponding to the application interface currently displayed by the mobile terminal, and the application category represents the functional characteristics of the target application. In other words, the target application is an application installed in the mobile terminal, which can be a built-in application of the mobile terminal, such as SMS, call and other applications, or a third-party application installed by the mobile terminal, such as video software, map software, etc., which is not limited in this embodiment of the present application. In addition, the functional characteristics represent the specific functions and services that the target application can provide to the user, and the target frame rate is the screen recording frame rate set for the functional characteristics and usage scenarios of the target application. This screen recording frame rate can be adapted to the functional characteristics and usage scenarios of the application. For details, please refer to the above description and will not be repeated here.

[0049] Specifically, the application category to which the target application belongs and the corresponding target frame rate may include but are not limited to the following situations:

[0050] ① The first category, the applications included in the first category are video playback applications; the target frame rate corresponding to the first category is the first frame rate value. For example, the video playback application can be a video software like "Youku" or "iQiyi", or a software with video playback function like "WeChat" or "Weibo", or it can be the video playback application 1011 in the above description, which is not limited in this application. It is understandable that in order to ensure that users have a good user experience when watching videos, the frame rate corresponding to the video application needs to reach 25 frames per second or more. Specifically, the first frame rate value can be 25, 26 or other similar frame rate values, which is not limited in this application.

[0051] ② The second category, the applications included in the second category are navigation applications; the target frame rate corresponding to the second category is a second frame rate value. The target frame rate corresponding to the second category is a second frame rate value. For example, the video playback application can be a third-party map software such as "AutoNavi Map" and "Baidu Map", or a web version of a navigation or map application, or it can be the video playback application 1012 in the above description, and this application does not limit this. It is understandable that when users use navigation applications, they do not pay as much attention to navigation applications as they do to video playback applications. Therefore, for navigation applications, a frame rate of 1 to 5 frames per second is sufficient to meet user needs in most cases. Therefore, the second frame rate value can be 4, 5 or other similar frame rate values, and this application does not limit this.

[0052] ③ Other categories, the applications included in the other categories are applications that are neither in the first category nor in the second category; the target frame rate corresponding to the other categories is a third frame rate value. It should be understood that, in addition to video playback applications and navigation applications, for other applications, such as music playback software or built-in call applications on mobile phones, or the music playback applications 1013 or call applications 1014 in the aforementioned description, etc. Most of the time, users may not pay attention to the screen information of these applications. Therefore, for these applications, the frame rate of the video only needs 1 to 2 frames per second when projecting the screen to meet the needs of users. Specifically, the third frame rate can be 4, 5 or other similar frame rate values, which is not limited in this application.

[0053] In this way, the screen recording frame rate of the mobile terminal can be set in a targeted manner according to the specific category and function of the application, the amount of data transmitted from the mobile terminal to the vehicle-mounted terminal can be greatly reduced, and the power consumption of the mobile terminal will also be effectively reduced.

[0054] In an optional embodiment, before performing step 201, the mobile terminal may also obtain the screen projection parameters input by the user, and the screen projection parameters are used to set the categories of multiple applications in the mobile terminal, and obtain the classification information of the multiple applications, and the classification information includes the application category and the target frame rate corresponding to the application category. It is understandable that different users have different personal preferences. For different users, the types of applications in their mobile terminals are different, and different users have different usage habits for the same application; for example, some users may be accustomed to frequently checking navigation routes. For such users, a frame rate of 1 to 5 frames per second may not be sufficient to meet the needs of users. Therefore, in this embodiment, the user can classify the applications in the mobile terminal in advance and set the corresponding frame rate for each category. Therefore, in this embodiment, in the subsequent screen projection process, the mobile terminal can obtain the category and frame rate corresponding to the application according to the currently displayed application, and perform screen recording according to the frame rate. The frame rate value obtained here is more suitable for the user's personal needs and can bring a better user experience to the user.

[0055] In an optional embodiment, during the screen recording process, the mobile terminal can also determine whether it has received the first information sent by the vehicle-mounted terminal, the first information indicating that the user is currently performing a touch operation on the screen projection interface, and the screen projection interface is the display interface of the vehicle-mounted terminal; when the first information is received, the mobile terminal can determine the target frame rate as the fourth frame rate value.

[0056] Because in the actual screen projection process, the user may make calls, switch songs, switch applications and other operations. At this time, the user often needs to touch the screen projection interface (that is, the display of the vehicle terminal) to complete these operations. At this time, in order to achieve smooth operation, it is necessary to ensure that the screen projection frame rate is large enough, for example, the frame rate needs to be greater than 15 frames per second. When the user touches the screen projection interface, the vehicle terminal will send a first message to the mobile terminal, and the first message indicates that the user is currently touching the screen projection interface. When the mobile terminal receives the first message, the mobile terminal determines the target frame rate as the fourth frame rate value.

[0057] In an optional embodiment, the mobile terminal will determine whether it has received second information sent by the vehicle-mounted terminal, the second information indicating that the user is not currently looking at the projection interface, and the projection interface is the display interface of the vehicle-mounted terminal; if the second information is received, the mobile terminal will interrupt the projection process.

[0058] During the actual driving of the vehicle, even if the mobile terminal is always projecting the screen to the vehicle terminal, the user may not always be looking at the screen. Therefore, in this embodiment, the vehicle terminal can use an eye tracking algorithm to determine whether the user is currently looking at the display of the vehicle terminal (i.e., the projection interface). When the user is not currently looking at the projection interface, the vehicle terminal will send a second message to the mobile terminal. When the mobile terminal receives the second message, the mobile terminal can temporarily interrupt the projection process.

[0059] It should be noted that the mobile terminal can interrupt the screen projection process by stopping screen recording; in addition, the mobile terminal can also interrupt the screen projection process by maintaining the screen recording action but stopping sending the recorded video to the vehicle-mounted terminal. This application does not limit this.

[0060] Correspondingly, the mobile terminal can also determine whether it has received the third information sent by the vehicle-mounted terminal, and the third information indicates that the user is currently looking at the screen projection interface; when the third information is received, the screen projection process is resumed.

[0061] Specifically, the action of determining the application category to which the target application belongs and the target frame rate corresponding to the application category can occur during the screen projection process. For example, when the user switches the application opened by the mobile terminal, the mobile terminal needs to re-determine the application category to which the newly opened application belongs and the target frame rate corresponding to the application category, and change the frame rate of the screen recording in time. In addition, this action can also occur at the beginning of the entire screen projection process. For example, after the mobile terminal and the vehicle-mounted terminal establish a communication connection, the mobile terminal can determine which application is open on the current interface of the mobile terminal before projection (that is, which application's user interface is displayed on the current interface, and the application is the target application at this time), and after obtaining the application category to which the application belongs and the target frame rate corresponding to the application category, the screen is recorded according to the target frame rate.

[0062] 202. Perform screen recording at the target frame rate to obtain a target video;

[0063] After determining the target frame rate, the mobile terminal will record according to the target frame rate, at the specified bit rate, resolution and duration, and perform video encoding on the recorded file to obtain the target video. Specifically, the target video can be in MPEG, AVI, MOV and other formats, which is not limited in the present embodiment.

[0064] 203. Send the target video to a vehicle-mounted terminal, where the vehicle-mounted terminal is used to play the target video.

[0065] In an embodiment of the present application, the vehicle-mounted terminal can be a vehicle-mounted device with data receiving and sending functions (such as a vehicle-mounted unit), a wireless terminal in unmanned driving, a terminal in transportation safety, a terminal in a smart city, a terminal device in a 5G network, or a terminal device in a public land mobile communication network that will evolve in the future, etc.

[0066] Optionally, when the vehicle-mounted terminal is a vehicle-mounted device in a vehicle, the vehicle may be an ordinary vehicle, a special vehicle (including but not limited to a police car, a tractor, etc.) or a rescue vehicle (including but not limited to an ambulance, a fire truck, a rescue vehicle, etc.). It is understandable that this application does not limit the specific form of the terminal device. Any device that can communicate with a roadside device, a vehicle, or a vehicle management platform, etc., falls within the protection scope of the terminal device.

[0067] The vehicle-mounted terminal and the mobile terminal can be connected by wired communication via a data cable or the like, or can be connected by wireless communication via a wireless local area network such as WIFI or Bluetooth.

[0068] It should be understood that during the entire screen projection process in the embodiment of the present application, although the mobile terminal still needs to keep the screen on, this method greatly reduces the video recording and video encoding work performed by the mobile terminal, thereby reducing the consumption of computing power of the CPU, GPU and codec module. In addition, this method can also effectively control the amount of data transmitted between the vehicle-mounted terminal and the mobile terminal, which can effectively reduce the consumption of power on the mobile terminal.

[0069] Based on the above method, the embodiment of the present application provides a schematic diagram of a scenario for screen projection based on a screen projection system. Figure 3 .

[0070] like Figure 3 As shown, Figure 2 The screen projection method in the embodiment can be completed based on the screen projection system 30. The screen projection system 30 includes: a mobile terminal 301 and a vehicle-mounted terminal 302.

[0071] The mobile terminal 301 includes a first display, and the vehicle-mounted terminal includes a second display; the mobile terminal 301 is used to determine the application category to which the target application belongs and the target frame rate corresponding to the application category, and to perform screen recording of the interface displayed on the first display at the target frame rate to obtain a target video, and send the target video to the vehicle-mounted terminal 302; the target application is the application corresponding to the application interface currently displayed by the mobile terminal 301, and the application category represents the functional characteristics of the target application.

[0072] The vehicle-mounted terminal 302 is used to receive the target video sent by the mobile terminal 301 and use the second display to play the target video.

[0073] The mobile terminal 301 is installed with a variety of built-in application software and other third-party application software, including: video playback application 3011, which can be used to play videos stored in the mobile phone or access and play video files on the Internet through the mobile network; navigation application 3012, which can be used for location search and route query functions, while providing real-time voice navigation, and can also display real-time itinerary route maps and real-time intersection details for users; music playback application 3013, which can be used to play various music files, and supports fast forward (previous song), fast rewind (next song), lyrics display, volume adjustment, playback background, spectrum display, search and other operations. Call application 3014, which can be used to make remote calls with others, and supports dialing, number location query, hang up and contact storage and other functions.

[0074] Before screen projection, user 303 can classify the applications in mobile terminal 302 according to his personal preferences and set the corresponding frame rate for each category. In this way, in the subsequent screen projection process, the mobile terminal will obtain the category and frame rate corresponding to the application according to the currently displayed application, and record the screen according to the frame rate, bringing a better user experience to the user.

[0075] like Figure 3 As shown, user 303 can classify the mobile terminals 302 into three categories, including:

[0076] ① The first category includes applications such as video playback applications (e.g. Figure 3 The target frame rate corresponding to the first category is a first frame rate value. Specifically, the first frame rate value may be 25 frames per second.

[0077] ② The second category, the applications included in the second category are navigation applications (such as Figure 3 The target frame rate corresponding to the second category is a second frame rate value. The target frame rate corresponding to the second category is a second frame rate value. Specifically, the second frame rate value may be 5 frames per second.

[0078] ③ Other categories: the applications included in the other categories are applications that are neither in the first category nor in the second category (e.g. Figure 3 The target frame rate corresponding to the other categories is a third frame rate value. Specifically, the third frame rate value may be 1 frame / second.

[0079] In addition, the screen of the vehicle-mounted terminal 302 is a touch screen 3021, which can sense the touch action performed by the user 303 on it (here, the touch action of the user 303 on the touch screen 3021 is collectively referred to as user behavior 3031). Furthermore, the vehicle-mounted terminal 302 is also provided with a function for capturing the facial image of the user 303, and the vehicle-mounted terminal 302 sends the captured image to the vehicle-mounted terminal 302 data processing center, and uses eye tracking technology to locate and track the face and eyes in the image, thereby estimating the line of sight of the user 303 (here, the action that the user's current line of sight falls on the touch screen 3021 (i.e., the second display) is referred to as user behavior 3032).

[0080] It should be understood that in the actual screen projection process, user 303 may make calls, switch songs, switch applications, etc. At this time, user 303 often needs to touch the screen projection interface (that is, the display of the vehicle terminal 302) to complete these operations. At this time, in order to achieve smooth operation, it is necessary to ensure that the screen projection frame rate is large enough, for example, the frame rate needs to be greater than 15 frames / second. Therefore, when user behavior 3031 occurs (that is, when user 303 touches the screen projection interface), the vehicle terminal 302 will send a first message to the mobile terminal 301, and the first information indicates that the user is currently touching the screen projection interface. When the mobile terminal 301 receives the first information, the mobile terminal 301 determines the target frame rate as the fourth frame rate value. Specifically, the fourth frame rate value can be 15 frames / second.

[0081] It should be understood that during the actual driving of the vehicle, even if the mobile terminal 301 is always projecting the screen to the vehicle terminal 302, the user may not always be looking at the screen. Therefore, in this embodiment, the vehicle terminal 302 can use an eye tracking algorithm to determine whether the user is currently looking at the display of the vehicle terminal 302. When the user behavior 3033 occurs (that is, when the user is not currently looking at the projection interface, Figure 3 (not shown), the vehicle terminal 302 will send the second information to the mobile terminal 301. When the mobile terminal 301 receives the second information, the mobile terminal 301 can temporarily interrupt the screen projection process. It should be noted that the mobile terminal 301 can interrupt the screen projection process by stopping the screen recording; in addition, the mobile terminal 301 can also interrupt the screen projection process by keeping the screen recording action but stopping sending the recorded video to the vehicle terminal 302. This application does not limit this.

[0082] Correspondingly, in this embodiment, the mobile terminal 301 can also determine whether the third information sent by the vehicle terminal is received. When the user behavior 3031 occurs (that is, when the user is currently looking at the projection interface), the vehicle terminal 302 will send the third information to the mobile terminal 3031 to indicate that the user is currently looking at the projection interface; when the third information is received, the mobile terminal 301 will resume the projection process.

[0083] The screen projection system 30 not only sets the screen recording frame rate of the mobile terminal 301 during the screen projection process in a targeted manner according to the specific category and function of the application, but also supports reasonably setting the frame rate of the mobile terminal when performing screen recording according to the user behavior of the user 303. This can greatly reduce the amount of data transmitted from the mobile terminal 301 to the vehicle-mounted terminal, effectively reduce the power consumption of the mobile terminal, and more intelligently provide users with projection images that can meet their real-time needs, saving system resources while providing users with a better user experience.

[0084] The following is a schematic diagram of the structure of a screen projection device provided in an embodiment of the present application. Figure 4 . Figure 4 The screen projection device in can perform Figure 2 The process of the screen projection method is as follows: Figure 4 As shown, the device may include:

[0085] A determination unit 401 determines an application category to which a target application belongs and a target frame rate corresponding to the application category; the target application is an application corresponding to an application interface currently displayed on the mobile terminal, and the application category represents functional characteristics of the target application; a recording unit 402 performs screen recording at the target frame rate to obtain a target video; a sending unit 403 sends the target video to a vehicle-mounted terminal, and the vehicle-mounted terminal is used to play the target video.

[0086] In an optional implementation, the determination unit 401 is specifically used to: determine whether the application category is a first category, and the applications included in the first category are video playback applications; when the application category is the first category, determine the target frame rate as a first frame rate value.

[0087] In an optional implementation, the determination unit 401 is specifically used to: determine whether the application category is the second category, and the applications included in the second category are navigation applications; when the application category is the second category, determine the target frame rate as the second frame rate value.

[0088] In an optional embodiment, the determination unit 401 is specifically used to: determine whether the application category is the first category or the second category, the applications included in the first category are video playback applications; the applications included in the second category are navigation applications; when the application category is not the first category and the application category is not the second category, determine the target frame rate as a third frame rate value.

[0089] In an optional embodiment, the device also includes: an input unit 404, used to obtain screen projection parameters input by a user, the screen projection parameters are used to set the categories of multiple applications in the mobile terminal, and obtain classification information of the multiple applications, the classification information including the application category and the target frame rate corresponding to the application category.

[0090] In an optional embodiment, the device also includes: a judgment unit 405, used to determine whether the first information sent by the vehicle-mounted terminal is received, the first information indicating that the user is currently performing a touch operation on the projection interface, and the projection interface is the display interface of the vehicle-mounted terminal; the determination unit 401 is also used to determine the target frame rate as a fourth frame rate value when the first information is received.

[0091] In an optional embodiment, the judgment unit 405 is also used to determine whether second information sent by the vehicle-mounted terminal is received, and the second information indicates that the user is not currently looking at the projection interface, and the projection interface is the display interface of the vehicle-mounted terminal; the device also includes: a control unit 406, which is used to interrupt the projection process when the second information is received.

[0092] In an optional embodiment, the judgment unit 405 is also used to determine whether the third information sent by the vehicle-mounted terminal is received, and the third information indicates that the user is currently looking at the screen projection interface; the control unit 406 is also used to resume the screen projection process when the third information is received.

[0093] It should be understood that the division of the various components of the above projection device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. For example, the above components can be separately established processing components, or they can be integrated into the same chip for implementation. In addition, they can also be stored in the storage element of the controller in the form of program code, and called and executed by a certain processing element of the processor. The functions of the above components. In addition, the various components can be integrated together or implemented independently. The processing element here can be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the method or the above components can be completed by hardware integrated logic circuits in the processor element or instructions in software form. The processing element can be a general-purpose processor, such as a CPU, or it can be one or more integrated circuits configured to implement the above method, such as: one or more ASICs (application-specific integrated circuits), or one or more DSPs (digital signal processors, microprocessors), or one or more FPGAs (field-programmable gate arrays, field programmable gate arrays), etc.

[0094] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 50 includes a processor 501, a memory 502, a communication interface 503 and a display; the processor 501, the memory 502, the communication interface 503 and the display 504 are interconnected via a bus. The electronic device may be the screen projection device described above.

[0095] The memory 502 includes but is not limited to RAM (random access memory), ROM (read-only memory), EPROM (erasable programmable read only memory), or CDROM (compact disc read-only memory), and the memory 502 is used for related instructions and data. The communication interface 503 is used to receive and send data. Specifically, the communication interface 503 can implement Figure 4 The functions of the sending unit 403 and the input unit 404.

[0096] The processor 501 may be one or more CPUs (central processing units). When the processor 501 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Figure 5 Specifically, the processor 501 can implement Figure 4 The functions of the determination unit 401, the judgment unit 405 and the control unit 406 are shown in FIG.

[0097] The display 504 may be a screen with display function such as a central control screen or HUD in a vehicle. Specifically, the display 504, the memory 502 and the processor 501 may implement Figure 4 The function of the recording unit 402.

[0098] The processor 501 in the electronic device 50 is used to read the program code stored in the memory 502 and execute the screen projection method in the aforementioned embodiment.

[0099] In an embodiment of the present application, another computer-readable storage medium is provided, which stores a computer program, which, when executed by a processor, implements: determining an application category to which a target application belongs and a target frame rate corresponding to the application category; the target application is an application corresponding to an application interface currently displayed on the mobile terminal, and the application category characterizes functional characteristics of the target application; performing screen recording at the target frame rate to obtain a target video; and sending the target video to a vehicle-mounted terminal, which is used to play the target video.

[0100] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the screen projection method provided in the aforementioned embodiment.

[0101] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described by flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0103] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A screen projection method, characterized in that: The method is applied to a mobile terminal and includes: Determine an application category to which a target application belongs and a target frame rate corresponding to the application category; the target application is an application corresponding to an application interface currently displayed by the mobile terminal, and the application category represents functional characteristics of the target application; Perform screen recording at the target frame rate to obtain a target video; Sending the target video to a vehicle-mounted terminal, wherein the vehicle-mounted terminal is used to play the target video; Set the target frame rate to different values ​​based on user behavior.

2. According to the method of claim 1, the step of determining the application category to which the target application belongs and the target frame rate corresponding to the application category comprises: Determine whether the application category is a first category, where applications included in the first category are video playback applications; When the application category is the first category, the target frame rate is determined to be a first frame rate value.

3. According to the method of claim 1, the step of determining the application category to which the target application belongs and the target frame rate corresponding to the application category comprises: Determining whether the application category is a second category, where the applications included in the second category are navigation applications; When the application category is the second category, the target frame rate is determined to be a second frame rate value.

4. According to the method of claim 1, the step of determining the application category to which the target application belongs and the target frame rate corresponding to the application category comprises: Determine whether the application category is a first category or a second category, and the applications included in the first category are video playback applications; The applications included in the second category are navigation applications; When the application category is not the first category and the application category is not the second category, the target frame rate is determined to be a third frame rate value.

5. According to the method according to any one of claims 1 to 4, before determining the application category and the target frame rate corresponding to the application category, the method further includes: Acquire screen projection parameters input by a user, wherein the screen projection parameters are used to set categories of multiple applications in the mobile terminal, and obtain classification information of the multiple applications, wherein the classification information includes the application category and the target frame rate corresponding to the application category.

6. The method according to any one of claims 1 to 4, further comprising: Determining whether first information sent by the vehicle-mounted terminal is received, the first information indicating that the user is currently performing a touch operation on a projection interface, the projection interface being a display interface of the vehicle-mounted terminal; When the first information is received, the target frame rate is determined as a fourth frame rate value.

7. The method according to any one of claims 1 to 5, further comprising: Determining whether second information sent by the vehicle-mounted terminal is received, the second information indicating that the user is not currently looking at the projection interface, the projection interface being the display interface of the vehicle-mounted terminal; When the second information is received, the screen projection process is interrupted.

8. The method according to claim 7, after interrupting the screen projection process, the method further comprises: Determining whether third information sent by the vehicle-mounted terminal is received, the third information indicating that the user is currently looking at the projection interface; When the third information is received, the screen projection process is resumed.

9. A screen projection system, characterized in that: The invention comprises a mobile terminal and a vehicle-mounted terminal, wherein the mobile terminal comprises a first display and the vehicle-mounted terminal comprises a second display; The mobile terminal is used to determine the application category to which the target application belongs and the target frame rate corresponding to the application category, and to perform screen recording of the interface displayed on the first display at the target frame rate to obtain a target video, and send the target video to the vehicle-mounted terminal; The target application is an application corresponding to the application interface currently displayed by the mobile terminal, and the application category represents the functional characteristics of the target application; The vehicle-mounted terminal is used to receive the target video sent by the mobile terminal and play the target video using the second display; The mobile terminal is also used to set the target frame rate to different values ​​according to user behavior.

10. An electronic device, characterized in that: include: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program is executed, the processor is configured to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is run on one or more processors, the method according to any one of claims 1 to 8 is executed.

Images