Screen projection method, device, equipment and storage medium
By generating and superimposing virtual screen video streams of car computers and mobile terminal devices, the user experience problem of car computers and mobile phone screen projection functions is solved, and the multi-channel video function is automatically supported, reducing update costs and time.
Patent Information
- Application Number
- CN202310151175.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-13
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-02-13
AI Technical Summary
The prior art is difficult to effectively realize the screen projection function between the car and the mobile phone, resulting in poor user experience, and updating the display device to support the multi-channel video function requires manual operation by users, which increases cost and time.
By obtaining the view content of the first virtual screen and the second virtual screen in the terminal device, a corresponding video stream is generated, and superimposed to generate a synthetic video stream and sent to the display device, automatic support of the multi-channel video function is realized.
It improves the user experience, realizes the multi-channel video function between the car and the mobile phone, reduces the need for users to update the display device, and saves costs and time.
Smart Images

Figure CN116193176B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to the field of vehicle networking technology. Background Art
[0002] As the development of automobiles becomes faster and faster, people have more and more demands for the interconnection between car computers and mobile phones. For example, a large number of users hope to connect car computers with mobile phones to realize the screen projection function. Among them, the screen projection function refers to projecting the content displayed on terminal devices such as mobile phones to the car computer. The screen projection function can enrich the ecology of car computers. Based on this, how to improve the screen projection function is an urgent problem to be solved. Summary of the invention
[0003] The present disclosure provides a screen projection method, apparatus, device, storage medium and program product.
[0004] According to one aspect of the present disclosure, a screen projection method is provided, including: obtaining a first video stream and a second video stream of a terminal device, wherein the first video stream is generated according to the view content of a first virtual screen in the terminal device, and the second video stream is generated according to the view content of a second virtual screen in the terminal device; superimposing the first video stream and the second video stream to obtain a composite video stream; and sending the composite video stream to a display device so that the display device displays the composite video stream.
[0005] According to another aspect of the present disclosure, a screen projection method is provided, including: generating a first video stream according to the view content of a first virtual screen; generating a second video stream according to the view content of a second virtual screen; and sending the first video stream and the second video stream to a conversion device.
[0006] According to another aspect of the present disclosure, a screen projection method is provided, including: receiving a composite video stream from a conversion device, wherein the composite video stream is obtained by superimposing a first video stream and a second video stream, the first video stream is generated according to the view content of a first virtual screen in a terminal device, and the second video stream is generated according to the view content of a second virtual screen in the terminal device; and displaying the composite video stream.
[0007] According to another aspect of the present disclosure, a screen projection device is provided, including: an acquisition module, used to acquire a first video stream and a second video stream of a terminal device, wherein the first video stream is generated according to the view content of a first virtual screen in the terminal device, and the second video stream is generated according to the view content of a second virtual screen in the terminal device; an overlay module, used to overlay the first video stream with the second video stream to obtain a composite video stream; and a first sending module, used to send the composite video stream to a display device so that the display device displays the composite video stream.
[0008] According to another aspect of the present disclosure, a screen projection device is provided, including: a first generating module, used to generate a first video stream according to the view content of a first virtual screen; a second generating module, used to generate a second video stream according to the view content of a second virtual screen; and a second sending module, used to send the first video stream and the second video stream to a conversion device.
[0009] According to another aspect of the present disclosure, a screen projection device is provided, including: a receiving module for receiving a composite video stream from a conversion device, wherein the composite video stream is obtained by superimposing a first video stream and a second video stream, the first video stream is generated according to the view content of a first virtual screen in a terminal device, and the second video stream is generated according to the view content of a second virtual screen in the terminal device; and a display module for displaying the composite video stream.
[0010] Another aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method shown in an embodiment of the present disclosure.
[0011] According to another aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method shown in the embodiment of the present disclosure.
[0012] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, wherein the computer program / instruction implements the steps of the method shown in the embodiment of the present disclosure when executed by a processor.
[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0015] Figure 1 is a schematic diagram of the system architecture of the screen projection method, device, electronic device and storage medium according to an embodiment of the present disclosure;
[0016] Figure 2 The flowchart of the screen projection method according to an embodiment of the present disclosure is schematically shown;
[0017] Figure 3 A flowchart of a screen projection method according to another embodiment of the present disclosure is schematically shown;
[0018] Figure 4 A flowchart of a screen projection method according to another embodiment of the present disclosure is schematically shown;
[0019] Figure 5 A flowchart schematically shows a conversion device acquiring a first video stream and a second video stream of a terminal device according to an embodiment of the present disclosure;
[0020] Figure 6 A flowchart schematically shows a method for acquiring a first video stream and a second video stream of a terminal device according to another embodiment of the present disclosure;
[0021] Figure 7 The flowchart of the method for obtaining a composite video stream according to an embodiment of the present disclosure is schematically shown;
[0022] Figure 8 A flowchart of a screen projection method according to another embodiment of the present disclosure is schematically shown;
[0023] Fig. 9 The flowchart of the method for obtaining a composite video stream according to an embodiment of the present disclosure is schematically shown;
[0024] Fig.10 A schematic diagram schematically shows a screen projection method according to an embodiment of the present disclosure;
[0025] Fig.11 A block diagram of a screen projection device according to an embodiment of the present disclosure is schematically shown;
[0026] Fig.12 A block diagram of a screen projection device according to another embodiment of the present disclosure is schematically shown;
[0027] Fig.13 A block diagram of a screen projection device according to another embodiment of the present disclosure is schematically shown;
[0028] Fig.14 A block diagram schematically illustrates an example electronic device that may be used to implement an embodiment of the present disclosure. DETAILED DESCRIPTION
[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0030] The following will be combined Figure 1 The system architecture of the screen projection method and device provided by the present invention is described.
[0031] Figure 1 is a schematic diagram of the system architecture of the screen projection method, device, electronic device and storage medium according to an embodiment of the present disclosure. It should be noted that, Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a display device 110, a terminal device 120 and a conversion device 130. The display device 110 may be used to display the picture data input to the display device 110. The terminal device 120 may include a display screen for displaying a local display interface. The conversion device 130 may include a CarLife conversion box, for example, where CarLife is a vehicle networking standard.
[0033] The display device 110 may be any electronic device having a display screen and supporting communication with other terminals, such as an in-vehicle infotainment device (also referred to as a vehicle computer). The terminal device 120 may be any electronic device having a display screen and supporting communication with other terminals, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. Various communication client applications may be installed on the display device 110 and the terminal device 120, such as an Internet of Vehicles application, a web browser application, a search application, an instant messaging tool, etc.
[0034] A wired or wireless connection may be established between the display device 110 and the conversion device 130. The wired connection may include, for example, a USB (Universal Serial Bus) connection, an HDMI (High Definition Multimedia Interface) connection, etc., and the wireless connection may include, for example, a WLAN (Wireless Local Area Network) connection, an NFC (Near Field Communication) connection, a Bluetooth connection, etc.
[0035] A wired or wireless connection may be established between the terminal device 120 and the conversion device 130. The wired connection may include, for example, a USB connection, an HDMI connection, etc., and the wireless connection may include, for example, a WLAN connection, an NFC connection, a Bluetooth connection, etc.
[0036] The terminal device 120 and the display device 110 can interact with each other through the conversion device 130. For example, the terminal device 120 can send the picture displayed on the display screen 121 to the conversion device 130 through the connection between the terminal device 120 and the conversion device 130. The conversion device 130 can send the picture from the terminal device 120 to the display device 110. After receiving the picture from the conversion device 130, the display device 110 can display the picture on the display screen 111 of the display device 110 to achieve screen projection. For another example, the user can perform touch operations and the like through the display device 110, and the display device 110 can send the operation information corresponding to the operation to the conversion device 130, and then the conversion device 130 sends the operation information to the terminal device 120. The terminal device 120 can perform corresponding operations according to the operation information.
[0037] Those skilled in the art will understand that Figure 1 The number of terminal devices and screen projection devices in the figure is only for illustration. Any number of terminal devices and screen projection devices may be provided according to implementation requirements.
[0038] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.
[0039] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.
[0040] The following will be combined Figure 2-4 The screen projection method provided by the present invention is described.
[0041] Figure 2 The flowchart of the screen projection method according to an embodiment of the present disclosure is schematically shown. The screen projection method can be applied to the above-mentioned conversion device, for example.
[0042] like Figure 2 As shown, the screen projection method 200 includes obtaining a first video stream and a second video stream of a terminal device in operation S210.
[0043] According to an embodiment of the present disclosure, the first video stream is generated according to the view content of the first virtual screen in the terminal device, and the second video stream is generated according to the view content of the second virtual screen in the terminal device.
[0044] Then, in operation S220, the first video stream is superimposed with the second video stream to obtain a composite video stream.
[0045] According to an embodiment of the present disclosure, for example, the picture of the first video stream can be superimposed on the upper layer of the corresponding picture in the second video stream, or the picture of the second video stream can be superimposed on the upper layer of the corresponding picture in the second video stream to obtain a composite video stream.
[0046] According to another embodiment of the present disclosure, the pictures of the first video stream and the pictures of the second video stream may also be displayed side by side to obtain a composite video stream.
[0047] According to an embodiment of the present disclosure, the resolutions of the first video stream and the second video stream may be the same or different.
[0048] In operation S230 , the synthesized video stream is sent to a display device so that the display device displays the synthesized video stream.
[0049] According to an embodiment of the present disclosure, a wired or wireless connection may be pre-established between the conversion device and the display device. The wired connection may include, for example, a USB connection, an HDMI connection, etc., and the wireless connection may include, for example, a WLAN connection, an NFC connection, a Bluetooth connection, etc. Based on this, the conversion device may send the composite video stream to the display device through the pre-established connection. After receiving the composite video stream, the display device may play the composite video stream.
[0050] According to an embodiment of the present disclosure, the first video stream and the second video stream may correspond to the interfaces of two applications respectively. By transmitting the first video stream and the second video stream through a conversion device, the display device can simultaneously display the interfaces of multiple applications, that is, realize a multi-channel video function, thereby improving the user experience. For example, a navigation application interface can be displayed on one side while a news application interface is displayed on the other side. In addition, only the conversion device needs to be updated to enable the display device to support the multi-channel video function. Since the user does not need to update the display device, it saves money and time.
[0051] Figure 3 The flowchart of a screen projection method according to another embodiment of the present disclosure is schematically shown. The screen projection method can be applied to the above-mentioned terminal device, for example.
[0052] like Figure 3 As shown, the screen projection method 300 includes, in operation S340, generating a first video stream according to the view content of the first virtual screen.
[0053] According to an embodiment of the present disclosure, the terminal device may pre-establish a first virtual screen (Display), then obtain the view content of the first virtual screen, and then encode the view content to obtain a first video stream.
[0054] In operation S350, a second video stream is generated according to the view content of the second virtual screen.
[0055] According to an embodiment of the present disclosure, the terminal device may also pre-establish a second virtual screen, then obtain the view content of the second virtual screen, and then encode the view content to obtain the first video stream.
[0056] In operation S360, the first video stream and the second video stream are transmitted to the conversion device.
[0057] According to an embodiment of the present disclosure, a wired or wireless connection may be pre-established between the terminal device and the conversion device. The wired connection may include, for example, a USB connection, an HDMI connection, etc., and the wireless connection may include, for example, a WLAN connection, an NFC connection, a Bluetooth connection, etc. Based on this, the terminal device may send the first video stream and the second video stream to the conversion device through the pre-established connection.
[0058] According to an embodiment of the present disclosure, the terminal device may display the first virtual screen and the second virtual screen. Alternatively, only one of the first virtual screen and the second virtual screen may be displayed, and the other may be hidden in the background. Alternatively, both the first virtual screen and the second virtual screen may be hidden in the background and not displayed.
[0059] Figure 4 The flowchart of a screen projection method according to another embodiment of the present disclosure is schematically shown. The screen projection method can be applied to the above-mentioned display device, for example.
[0060] like Figure 4 As shown, the screen projection method 400 includes, in operation S470, receiving a composite video stream from a conversion device.
[0061] According to an embodiment of the present disclosure, the display device may receive the composite video stream through a connection with the conversion device, for example.
[0062] In operation S480, the synthesized video stream is presented.
[0063] According to an embodiment of the present disclosure, after receiving the composite video stream, the display device may, for example, play the composite video stream to display the composite video stream.
[0064] The following will be combined Figure 5 A method for a conversion device provided in the present disclosure to obtain a first video stream and a second video stream of a terminal device is described.
[0065] Figure 5 The flowchart of a conversion device acquiring a first video stream and a second video stream of a terminal device according to an embodiment of the present disclosure is schematically shown.
[0066] like Figure 5 As shown, the method 500 for the conversion device to obtain the first video stream and the second video stream of the terminal device includes: in operation S510, the terminal device creates a first transmission channel and a second transmission channel.
[0067] According to an embodiment of the present disclosure, the first transmission channel may include, for example, a Video Socket (video channel).
[0068] In operation S520, the terminal device sends a first video stream to the conversion device through a first transmission channel.
[0069] According to an embodiment of the present disclosure, the conversion device may, for example, be pre-connected to the first transmission channel, so that the terminal device may transmit the first video stream to the conversion device through the first transmission channel.
[0070] In operation S530, the conversion device acquires a first video stream from the terminal device through a first transmission channel.
[0071] In operation S540, the terminal device transmits the second video stream to the conversion device through the second transmission channel.
[0072] According to an embodiment of the present disclosure, the second transmission channel may include, for example, a Video Socket.
[0073] In operation S550, the conversion device acquires a second video stream from the terminal device through the second transmission channel.
[0074] According to an embodiment of the present disclosure, the conversion device may, for example, be pre-connected to the second transmission channel, so that the terminal device may transmit the second video stream to the conversion device through the second transmission channel.
[0075] According to the embodiments of the present disclosure, by transmitting through the first and second channels, the logic is clear from the business layer and is easy for developers to understand. The first transmission channel transmits the first video stream, and the second transmission channel transmits the second video stream, and there is no need to distinguish between the first video stream and the second video stream.
[0076] The following will be combined Figure 6 Another method for a conversion device provided by the present disclosure to obtain a first video stream and a second video stream of a terminal device is described.
[0077] Figure 6 A flowchart of a method for acquiring a first video stream and a second video stream of a terminal device according to another embodiment of the present disclosure is schematically shown.
[0078] like Figure 6 As shown, the method 600 for acquiring the first video stream and the second video stream of the terminal device includes, in operation S610, the terminal device creates a third transmission channel.
[0079] According to the disclosed embodiment, the third transmission channel may include, for example, a Video Socket.
[0080] In operation S620, the terminal device generates original video data according to the first video stream and the second video stream.
[0081] According to an embodiment of the present disclosure, for example, the first video stream and the second video stream can be encapsulated together to obtain original video data. The original video data may include an identifier, and the identifier is used to indicate the position of the first video stream and the second video stream in the original video data. Exemplarily, the identifier can be recorded in the header of the original video data.
[0082] In operation S630, the terminal device transmits the first video stream to the conversion device through the third transmission channel.
[0083] In operation S640, the conversion device acquires original video data from the terminal device through the third transmission channel.
[0084] In operation S650, the conversion device extracts the first video stream and the second video stream from the original video data according to the identifier in the original video data.
[0085] According to an embodiment of the present disclosure, the conversion device can determine the position of the first video stream and the position of the second video stream in the original video data according to the identifier, and extract the first video stream and the second video stream from these positions.
[0086] According to the embodiments of the present disclosure, by transmitting the first video stream and the second video stream through one transmission channel, resources such as memory and CPU (central processing unit) can be saved.
[0087] The following will be combined Figure 7 The method for obtaining a composite video stream provided by the present disclosure is described.
[0088] Figure 7 The flowchart of the method for obtaining a composite video stream according to an embodiment of the present disclosure is schematically shown.
[0089] like Figure 7 As shown, the method 720 for obtaining a composite video stream includes generating a first view component according to a first video stream in operation S721.
[0090] According to an embodiment of the present disclosure, the first video component may include, for example, a SurfaceView component. SurfaceView is an inherited class of View, and a Surface dedicated to drawing is embedded in SurfaceView. Surface is a handle to a native buffer managed by a screen compositor.
[0091] For example, the first video stream may be decoded, and the decoded video stream may be rendered into a SurfaceView component to obtain a first video component.
[0092] In operation S722, a second view component is generated according to the second video stream.
[0093] According to an embodiment of the present disclosure, the second video component may include, for example, a SurfaceView component. For example, the second video stream may be decoded, and the decoded video stream may be rendered into the SurfaceView component to obtain the second video component.
[0094] According to an embodiment of the present disclosure, operation S721 and operation S722 may be performed in any order.
[0095] In operation S723, the first view component and the second view component are placed in a stacked manner to obtain a target view.
[0096] According to an embodiment of the present disclosure, for example, the first view component may be stacked on an upper layer of the second view component, or the second view component may be stacked on an upper layer of the first view component.
[0097] In operation S724 , a synthesized video stream is determined according to the target view.
[0098] According to an embodiment of the present disclosure, for example, a target view may be captured at a predetermined frequency to obtain multiple images, and then a composite video stream is generated based on the multiple images, wherein the predetermined frequency may be set according to actual needs.
[0099] For example, the conversion device can capture the view of the entire SurfaceView area at a frequency of 30 frames per second to obtain a corresponding image, and then soft-encode the image to obtain encoded binary data, that is, a synthetic video stream. The soft encoding can be implemented based on ffmpeg, for example.
[0100] Figure 8 A flowchart of a screen projection method according to another embodiment of the present disclosure is schematically shown.
[0101] like Figure 8 As shown, the screen projection method 800 may further include, in operation S880, the display device may determine coordinate information of the control operation in response to receiving a control instruction for the composite video stream.
[0102] According to an embodiment of the present disclosure, coordinate information may be used to indicate the location of a control operation. A user may input a control instruction for a composite video stream to a display device through a display device. For example, a user may perform a touch operation in a display device, generate a corresponding control instruction, and obtain coordinate information corresponding to the touch operation. The coordinate information may include, for example, (x, y), where x may be a horizontal axis coordinate and y may be a vertical axis coordinate.
[0103] In operation S890, the display device transmits the coordinate information to the conversion device.
[0104] In operation S8100, the conversion device transmits the coordinate information to the terminal device.
[0105] In operation S8110, the terminal device receives coordinate information from the conversion device, and performs an operation corresponding to the first virtual screen or the second virtual screen according to the coordinate information.
[0106] According to an embodiment of the present disclosure, the operation may include, for example, application startup or shutdown, function startup or shutdown, system setting, etc.
[0107] For example, the location indicated by the coordinate information may be a start button of application A, and an operation of triggering the button may be performed accordingly to start application A.
[0108] According to another embodiment of the present disclosure, the conversion device may also set a manipulation area and a content area in the target view. Then the first view component or the second view component is placed in the content area. Next, in response to receiving coordinate information from the display device, the coordinate information may be sent to the terminal device when the coordinate information matches the content area. When the coordinate information matches the manipulation area, the position of the content area is adjusted according to the coordinate information. Exemplarily, if the position indicated by the coordinate information is within the content area, it is determined that the coordinate information matches the content area, otherwise, it is determined that the coordinate information does not match the content area.
[0109] Reference below Fig. 9 The above-mentioned composite video stream is further described in conjunction with a specific embodiment. Those skilled in the art can understand that the following exemplary embodiments are only used to understand the present disclosure, and the present disclosure is not limited thereto.
[0110] Fig. 9The flowchart of the method for obtaining a composite video stream according to an embodiment of the present disclosure is schematically shown.
[0111] like Fig. 9 As shown, exemplarily, in this embodiment, the first video stream can be a vertical screen video stream (a video stream with a horizontal resolution smaller than a vertical resolution), and the second video stream can be a horizontal screen video stream (a video stream with a vertical resolution smaller than a horizontal resolution). Based on this, the conversion device can generate a first video component 901 according to the first video stream, and generate a second video component according to the second video stream. Then, the first video component 901 can be superimposed on the upper layer of the second view component 902. The conversion device can also generate a control area 903. When the user drags the control area 903, the conversion device can adjust the position of the content area 904 accordingly according to the coordinate information of the dragging operation. For example, if the user drags the control area 903 horizontally, the conversion device can move the content area 904 horizontally accordingly.
[0112] According to another embodiment of the present disclosure, the conversion device may pre-acquire the resolution of the display device. Based on this, the terminal device may also acquire the resolution of the display device through the conversion device. According to the resolution, the first virtual screen and the second virtual screen are generated. For example, the vertical resolution of the first virtual screen may be set according to the vertical resolution of the display device, and the horizontal resolution of the first virtual screen may be set according to the video ratio and vertical resolution corresponding to the first virtual screen. Alternatively, the horizontal resolution of the second virtual screen may be set according to the horizontal resolution of the display device, and the vertical resolution of the second virtual screen may be set according to the video ratio and horizontal resolution corresponding to the second virtual screen.
[0113] Reference below Fig.10 , the screen projection method shown above is further described in conjunction with a specific embodiment. Those skilled in the art can understand that the following exemplary embodiments are only used to understand the present disclosure, and the present disclosure is not limited thereto.
[0114] Fig.10 A schematic diagram of a screen projection method according to an embodiment of the present disclosure is schematically shown.
[0115] exist Fig.10 As shown in FIG. 1 , in operation S1001 , the conversion device and the display device may establish a wired connection. For example, the conversion device and the display device may be connected via a USB.
[0116] In operation S1002, the conversion device and the terminal device may establish a wireless connection. For example, the terminal device may detect a Bluetooth signal of the conversion device and connect to the conversion device according to the Bluetooth signal.
[0117] After the conversion device and the display device, as well as the conversion device and the terminal device are successfully connected, the conversion device notifies the terminal device to enable the multi-channel video function.
[0118] Then in operation S1003, the terminal device may start transmitting a vertical video stream, ie, a first video stream, through the Video Socket A channel.
[0119] Before transmission, the terminal device can create a new vertical virtual screen (Display) according to the agreed resolution. Initially, the virtual screen displays the initial page, which can have multiple icons (icons) corresponding to multiple applications. The user can select the target application from multiple applications. After the user selects the target application, the virtual screen will display the interface of the target application.
[0120] Then the terminal device can monitor the page element changes of the vertical screen virtual screen through MediaCodec. Among them, MediaCodec is an audio and video encoding and decryption tool. When the page element changes are monitored, the view content of the vertical screen virtual screen is obtained, and then encoding is started to generate a video stream. Next, the terminal device monitors the transmission of the vertical screen video stream through the Video Socket A channel.
[0121] Similarly, the terminal device creates a new horizontal virtual screen at the agreed resolution. Then the terminal device can monitor the page element changes of the horizontal virtual screen through MediaCodec. After monitoring the page element changes, it starts to obtain the view content of the horizontal virtual screen, and then starts encoding to generate a video stream. Then in operation S1004, the vertical video stream, i.e., the second video stream, is transmitted through the VideoSocket B channel.
[0122] In operation S1005, after the conversion device establishes a connection with the terminal device, it can create two new SurfaceView components, one of which is a vertical screen SurfaceView component and the other is a horizontal screen SurfaceView component. The vertical screen SurfaceView component can be used to display the vertical screen video stream, and the horizontal screen SurfaceView component can be used to display the horizontal screen video stream. The vertical screen SurfaceView component can then be stacked on top of the horizontal screen SurfaceView component. After receiving the vertical screen video stream, the video stream will be decoded and then rendered into the vertical screen SurfaceView. After receiving the horizontal screen video stream, the video stream will be decoded and then rendered into the horizontal screen SurfaceView.
[0123] Next, the conversion device can capture the view of the entire SurfaceView area at a rate of 30 frames per second, and obtain a video stream, that is, a synthesized video stream, by soft-encoding the captured image through ffmpeg.
[0124] Then in operation S1006 , the conversion device may send the synthesized video stream to the display device in real time.
[0125] In operation S1007 , when the display device receives a touch signal, such as a click signal or a slide signal, the display device may send the real-time touch coordinates (x, y) to the conversion device.
[0126] After receiving the touch coordinates, the conversion device can determine whether the coordinates belong to the content area of the vertical screen or the content area of the horizontal screen. If so, in operation S1008, the coordinates are directly given to the terminal device, and the corresponding virtual screen in the terminal device directly consumes and reacts. The coordinates in the content area of the vertical screen are consumed by the vertical screen virtual screen, and the coordinates in the content area of the horizontal screen are consumed by the horizontal screen virtual screen.
[0127] The conversion device can also set a control area in the SurfaceView component of the vertical screen. Based on this, in operation S1009, if the received coordinates are the control area, it is consumed by the conversion device itself, and the conversion device adjusts the position of the SurfaceView component of the vertical screen according to the coordinates, so that the content area of the vertical screen can change position according to the touch.
[0128] The following will be combined Fig.11 The conversion device provided by the present disclosure is described.
[0129] Fig.11 The block diagram of a screen projection device according to an embodiment of the present disclosure is schematically shown. The screen projection device can be applied to a conversion device.
[0130] like Fig.11 As shown, the screen projection device 1100 includes an acquisition module 1110 , an overlay module 1120 and a first sending module 1130 .
[0131] The acquisition module 1110 is used to acquire the first video stream and the second video stream of the terminal device, wherein the first video stream is generated according to the view content of the first virtual screen in the terminal device, and the second video stream is generated according to the view content of the second virtual screen in the terminal device.
[0132] The superposition module 1120 is used to superimpose the first video stream and the second video stream to obtain a composite video stream.
[0133] The first sending module 1130 is used to send the synthesized video stream to a display device so that the display device displays the synthesized video stream.
[0134] According to an embodiment of the present disclosure, the overlay module may include: a first component generation submodule, used to generate a first view component according to a first video stream; a second component generation submodule, used to generate a second view component according to a second video stream; a stacking submodule, used to place the first view component and the second view component in a stacked manner to obtain a target view; and a synthesis submodule, used to determine a synthesized video stream according to the target view.
[0135] According to an embodiment of the present disclosure, the synthesis submodule may include: a screenshot unit, configured to screenshot the target view at a predetermined frequency to obtain a plurality of images; and a video stream generation unit, configured to generate a synthesized video stream based on the plurality of images.
[0136] According to an embodiment of the present disclosure, the screen projection device 1100 may further include: a setting module for setting a manipulation area and a content area in a target view; and a placement module for placing the first view component or the second view component in the content area.
[0137] According to an embodiment of the present disclosure, the projection device 1100 may further include: a forwarding module for sending the coordinate information to the terminal device in response to receiving the coordinate information from the display device when the coordinate information matches the content area; and an adjustment module for adjusting the position of the content area according to the coordinate information when the coordinate information matches the control area.
[0138] According to an embodiment of the present disclosure, the acquisition module may include: a first acquisition submodule, used to acquire a first video stream from a terminal device through a first transmission channel; and a second acquisition submodule, used to acquire a second video stream from the terminal device through a second transmission channel.
[0139] According to another embodiment of the present disclosure, the acquisition module may include: a third acquisition sub-module, used to acquire original video data from a terminal device through a third transmission channel; and an extraction sub-module, used to extract the first video stream and the second video stream from the original video data according to an identifier in the original video data, wherein the identifier is used to indicate the position of the first video stream and the second video stream in the original video data.
[0140] The following will be combined Fig.12 The terminal device provided by the present disclosure is described.
[0141] Fig.12 The block diagram of a screen projection device according to another embodiment of the present disclosure is schematically shown. The screen projection device can be applied to a terminal device.
[0142] like Fig.12As shown, the screen projection device 1200 includes a first generating module 1210, a second generating module 1220 and a second sending module 1230. The first generating module 1210 is used to generate a first video stream according to the view content of the first virtual screen.
[0143] The second generating module 1220 is used to generate a second video stream according to the view content of the second virtual screen.
[0144] The second sending module 1230 is configured to send the first video stream and the second video stream to the conversion device.
[0145] According to an embodiment of the present disclosure, the second sending module may include: a first creation submodule, used to create a first transmission channel and a first transmission channel; a first video stream sending submodule, used to send the first video stream to the conversion device through the first transmission channel; and a second video stream sending submodule, used to send the second video stream to the conversion device through the second transmission channel.
[0146] According to an embodiment of the present disclosure, the second sending module may include: a second creation submodule, used to create a third transmission channel; an original video data generation submodule, used to generate original video data based on the first video stream and the second video stream, wherein the original video data includes an identifier, which is used to indicate the position of the first video stream and the second video stream in the original video data; and a third video stream sending submodule, used to send the original video data to the conversion device through the third transmission channel.
[0147] According to an embodiment of the present disclosure, the screen projection device 1200 may further include: a resolution acquisition module, used to acquire the resolution of the display device through a conversion device; and a virtual screen generation module, used to generate a first virtual screen and a second virtual screen according to the resolution.
[0148] According to an embodiment of the present disclosure, the screen projection device 1200 may further include: a coordinate receiving module for receiving coordinate information from a conversion device; and an execution module for executing operations corresponding to the first virtual screen or the second virtual screen according to the coordinate information.
[0149] The following will be combined Fig.13 The screen projection device provided by the present invention is described.
[0150] Fig.13 A block diagram of a screen projection device according to another embodiment of the present disclosure is schematically shown. The screen projection device can be applied to a display device.
[0151] like Fig.13 As shown, the screen projection device 1300 includes a receiving module 1310 and a display module 1320 .
[0152] The receiving module 1310 is used to receive a composite video stream from a conversion device, wherein the composite video stream is obtained by superimposing a first video stream and a second video stream, wherein the first video stream is generated based on the view content of a first virtual screen in a terminal device, and the second video stream is generated based on the view content of a second virtual screen in the terminal device.
[0153] The display module 1320 is used to display the synthesized video stream.
[0154] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0155] Fig.14 A block diagram of an example electronic device 1400 that can be used to implement an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0156] like Fig.14 As shown, the device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.
[0157] A number of components in the device 1400 are connected to the I / O interface 1405, including: an input unit 1406, such as a keyboard, a mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a disk, an optical disk, etc.; and a communication unit 1409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the device 1400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0158] The computing unit 1401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above, such as the screen projection method. For example, in some embodiments, the screen projection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the screen projection method described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to perform the screen projection method in any other appropriate manner (e.g., by means of firmware).
[0159] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0160] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0161] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0163] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0164] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0165] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.
[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0167] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A screen projection method, comprising: Acquire a first video stream and a second video stream of a terminal device, wherein the first video stream is generated according to the view content of a first virtual screen in the terminal device, and the second video stream is generated according to the view content of a second virtual screen in the terminal device; Superimposing the first video stream and the second video stream to obtain a composite video stream, wherein the composite video stream includes a target view; and Sending the composite video stream to a display device so that the display device displays the composite video stream; Wherein, the target view includes a control area and a content area; and the method further includes: In response to receiving the coordinate information from the display device, if the coordinate information matches the content area, sending the coordinate information to the terminal device; and When the coordinate information matches the manipulation area, the position of the content area is adjusted according to the coordinate information.
2. The method according to claim 1, wherein: The step of superimposing the first video stream and the second video stream to obtain a composite video stream includes: Generate a first view component according to the first video stream; generating a second view component according to the second video stream; Placing the first view component and the second view component in a stacked manner to obtain a target view; and The synthesized video stream is determined according to the target view.
3. The method according to claim 2, wherein: The step of determining the synthesized video stream according to the target view includes: Taking screenshots of the target view at a predetermined frequency to obtain a plurality of images; and The composite video stream is generated based on the multiple images.
4. The method according to claim 2, further comprising: Place the first view component or the second view component in the content area.
5. The method according to claim 1, wherein: The obtaining of the first video stream and the second video stream of the terminal device comprises: Acquire the first video stream from the terminal device through a first transmission channel; and The second video stream is obtained from the terminal device through a second transmission channel.
6. The method according to claim 1, wherein: The obtaining of the first video stream and the second video stream of the terminal device comprises: Acquiring original video data from the terminal device via a third transmission channel; and The first video stream and the second video stream are extracted from the original video data according to an identifier in the original video data, wherein the identifier is used to indicate positions of the first video stream and the second video stream in the original video data.
7. A screen projection method, comprising: Generate a first video stream according to the view content of the first virtual screen; Generate a second video stream according to the view content of the second virtual screen; as well as sending the first video stream and the second video stream to a conversion device; Wherein, the method further comprises: receiving coordinate information from the conversion device; and According to the coordinate information, executing an operation corresponding to the first virtual screen or the second virtual screen; The conversion device superimposes the first video stream and the second video stream to obtain a composite video stream, wherein the composite video stream includes a target view, the target view includes a control area and a content area, and the coordinate information matches the content area.
8. The method according to claim 7, wherein: The sending the first video stream and the second video stream to the conversion device comprises: Creating a first transmission channel and a second transmission channel; Sending the first video stream to the conversion device through the first transmission channel; and The second video stream is sent to the conversion device through the second transmission channel.
9. The method according to claim 7, wherein: The sending the first video stream and the second video stream to the conversion device comprises: Create a third transmission channel; Generate original video data according to the first video stream and the second video stream, wherein the original video data includes an identifier, and the identifier is used to indicate the positions of the first video stream and the second video stream in the original video data; and The original video data is transmitted to the conversion device through the third transmission channel.
10. The method according to claim 7, further comprising: Acquiring the resolution of the display device through the conversion device; as well as The first virtual screen and the second virtual screen are generated according to the resolution.
11. A screen projection method, comprising: Receiving a composite video stream from a conversion device, wherein the composite video stream is obtained by superimposing a first video stream and a second video stream, and the composite video stream includes a target view; the first video stream is generated according to the view content of a first virtual screen in a terminal device, and the second video stream is generated according to the view content of a second virtual screen in the terminal device; and displaying the synthesized video stream; Wherein, the target view includes a control area and a content area; the conversion device responds to the received coordinate information, and when the coordinate information matches the content area, sends the coordinate information to the terminal device; and when the coordinate information matches the control area, adjusts the position of the content area according to the coordinate information.
12. The method according to claim 11, further comprising: In response to receiving a control instruction for the composite video stream, determining coordinate information of a control operation; as well as The coordinate information is sent to the conversion device.
13. A screen projection device, comprising: An acquisition module, configured to acquire a first video stream and a second video stream of a terminal device, wherein the first video stream is generated according to the view content of a first virtual screen in the terminal device, and the second video stream is generated according to the view content of a second virtual screen in the terminal device; a superposition module, configured to superimpose the first video stream and the second video stream to obtain a composite video stream, wherein the composite video stream includes a target view; and A first sending module, used for sending the composite video stream to a display device so that the display device displays the composite video stream; The target view includes a control area and a content area; the projection device also includes a module for: In response to receiving the coordinate information from the display device, if the coordinate information matches the content area, sending the coordinate information to the terminal device; and When the coordinate information matches the manipulation area, the position of the content area is adjusted according to the coordinate information.
14. A screen projection device, comprising: A first generating module, used to generate a first video stream according to the view content of the first virtual screen; A second generating module, used to generate a second video stream according to the view content of the second virtual screen; as well as A second sending module, used for sending the first video stream and the second video stream to a conversion device; The screen projection device further includes a module for: receiving coordinate information from the conversion device; and According to the coordinate information, executing an operation corresponding to the first virtual screen or the second virtual screen; The conversion device superimposes the first video stream and the second video stream to obtain a composite video stream, wherein the composite video stream includes a target view, the target view includes a control area and a content area, and the coordinate information matches the content area.
15. A screen projection device, comprising: a receiving module, configured to receive a composite video stream from a conversion device, wherein the composite video stream is obtained by superimposing a first video stream and a second video stream, the composite video stream includes a target view, the first video stream is generated according to a view content of a first virtual screen in a terminal device, and the second video stream is generated according to a view content of a second virtual screen in the terminal device; and A display module, used for displaying the synthesized video stream; Wherein, the target view includes a control area and a content area; the conversion device responds to the received coordinate information, and when the coordinate information matches the content area, sends the coordinate information to the terminal device; and when the coordinate information matches the control area, adjusts the position of the content area according to the coordinate information.
16. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.
17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.
18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
Citation Information
Patent Citations
Screen projection method and terminal
CN109992231A
Application screen projection control method and device, equipment and medium
CN111432070A
Vehicle machine screen display method, vehicle and computer readable storage medium
CN112118558A
Split-screen display method based on mobile phone interconnection
CN112202967A
Information sharing method and vehicle-mounted terminal
CN112214186A
