Mode switching method and system of audio and video data, computer device and computer readable storage medium

Abstract

This application relates to a method, system, computer device, and computer-readable storage medium for switching audio and video data modes. By responding to an access request to a target camera device, an audio and video connection is obtained; based on the audio and video connection, first audio and video data is displayed on a first page corresponding to a first mode; in response to a switching request, a second mode corresponding to the switching request is determined, and while maintaining the audio and video connection, a data request instruction corresponding to the second mode is sent to the target camera device through the audio and video connection; second audio and video data returned by the target camera device based on the data request instruction is obtained through the audio and video connection, and the second audio and video data is displayed on a second page corresponding to the second mode. This eliminates the need for device wake-up, session establishment, and waiting for the first frame data during the switching process, thus improving the efficiency of audio and video data switching.

Description

Technical Field

[0001] This application relates to the field of audio and video processing technology, and in particular to a method, system, computer device, and computer-readable storage medium for switching audio and video data modes. Background Technology

[0002] With the widespread adoption of IoT cameras in home security, commercial surveillance, and other fields, mobile applications have become the primary entry point for users to view camera footage. Users typically need to switch between live streaming and memory card recording playback modes during use; for example, viewing the current feed and then browsing historical recordings, or returning to live feeds during playback to confirm the latest situation.

[0003] In related technologies, live streaming and memory card playback each correspond to independent display pages, and audio and video connections are established and held separately by each display page. Therefore, when switching from the live streaming page to the playback page, the audio and video connection held by the current page must be disconnected first, and then the audio and video connection must be re-established on the playback page; the same applies when switching back from the playback page to the live streaming page.

[0004] In the above method, each mode switch is accompanied by the disconnection and re-establishment of the audio and video connection. The establishment of the audio and video connection requires the process of device wake-up, session establishment, transmission channel negotiation and waiting for the first frame of data. Before the process is completed, the terminal device has no audio and video data to display, the picture is interrupted, and the switching efficiency of audio and video data is low. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, system, computer device, and computer-readable storage medium for switching audio and video data modes that can improve the efficiency of audio and video data switching, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for switching audio and video data modes, applied to a terminal device communicating with a target camera device, the method comprising:

[0007] In response to an access request to the target camera device, an audio-visual connection is obtained; wherein, the audio-visual connection is a connection between the terminal device and the target camera device used to transmit audio-visual data, and the access request is used to request the playback of the first audio-visual data in the first mode;

[0008] Based on the audio and video connection, the first audio and video data is displayed on the first page corresponding to the first mode;

[0009] In response to a switching request, a second mode corresponding to the switching request is determined, and while maintaining the audio and video connection, a data request instruction corresponding to the second mode is sent to the target camera device through the audio and video connection; wherein, the first mode and the second mode are different audio and video data playback modes;

[0010] The second audio and video data is obtained by connecting the audio and video device to the target camera device based on the data request command, and then displayed on the second page corresponding to the second mode.

[0011] In one embodiment, the step of obtaining an audio / video connection in response to an access request to a target camera device includes:

[0012] In response to the access request, obtain the device identifier of the target camera device;

[0013] Check if a connection management unit corresponding to the device identifier exists; the connection management unit is used to maintain the audio and video connection corresponding to the device identifier.

[0014] In the absence of a connection management unit, a connection management unit corresponding to the device identifier is created, and an audio-visual connection is established with the target camera device through the connection management unit.

[0015] In the presence of a connection management unit, the connection currently maintained by the connection management unit is identified as an audio / video connection.

[0016] In one embodiment, the step of displaying the first audio / video data on the first page corresponding to the first mode based on the audio / video connection includes:

[0017] Set the first page as the current display page for the audio / video link;

[0018] Send the current connection status of the audio / video connection to the currently displayed page;

[0019] When the current connection status indicates that the audio and video connection is ready, establish a first binding relationship between the rendered view of the first page and the first audio and video data output by the audio and video connection;

[0020] Display the first audio and video data based on the first binding relationship.

[0021] In one embodiment, after the step of displaying the first audio and video data on a first page corresponding to the first mode based on the audio and video connection, the method further includes:

[0022] In response to an exit operation on the currently displayed page, the mapping between the currently displayed page and the audio / video connection is severed;

[0023] Set a destruction marker for the connection management unit and start a delayed destruction timer;

[0024] The audio and video connection is maintained until the pending destruction marker is cleared and the delayed destruction timer expires.

[0025] If the delayed destruction timer expires and the destruction marker is not cleared, the connection management unit is destroyed and the audio / video connection is disconnected.

[0026] If the connection management unit is found before the delayed destruction timer expires, the destruction marker is cleared and the audio / video connection is maintained.

[0027] In one embodiment, the step of determining a second mode corresponding to the switching request in response to the switching request, and sending a data request instruction corresponding to the second mode to the target camera device through the audio-video connection while maintaining the audio-video connection, includes:

[0028] In response to a switching request, while maintaining the audio and video connection, the first binding relationship between the rendered view of the first page and the first audio and video data output by the audio and video connection is released;

[0029] Change the current display page of the audio / video connection from the first page to the second page;

[0030] Update the audio / video connection mode marker from the first mode to the second mode;

[0031] Send a data request command corresponding to the second mode to the target camera device via audio / video connection.

[0032] In one embodiment, the step of displaying the second audio / video data on the second page corresponding to the second mode includes:

[0033] Send the current connection status of the audio / video connection to the second page;

[0034] If the current connection status indicates that the audio and video connection is ready, establish a second binding relationship between the rendered view of the second page and the second audio and video data output by the audio and video connection;

[0035] The second audio and video data is displayed based on the second binding relationship.

[0036] In one embodiment, the target camera device is a first camera device among a plurality of camera devices, and the plurality of camera devices further includes a second camera device; the method further includes:

[0037] The connection management units corresponding to each of the multiple camera devices are stored in the connection management unit pool according to the device identifier of each camera device; wherein, each connection management unit maintains the audio and video connection with the corresponding camera device.

[0038] When the audio / video connection corresponding to the first camera device is switched from the first mode to the second mode, the audio / video connection corresponding to the second camera device is maintained.

[0039] Secondly, this application also provides an audio / video data mode switching system, applied to a terminal device communicating with a target camera device, the system comprising:

[0040] The connection module is used to obtain an audio-visual connection in response to an access request to the target camera device; wherein, the audio-visual connection is a connection between the terminal device and the target camera device for transmitting audio-visual data, and the access request is used to request the playback of the first audio-visual data in the first mode;

[0041] The first display module is used to display first audio and video data on a first page corresponding to the first mode based on the audio and video connection.

[0042] The switching module is used to respond to a switching request, determine the second mode corresponding to the switching request, and send a data request command corresponding to the second mode to the target camera device through the audio and video connection while maintaining the audio and video connection; wherein, the first mode and the second mode are different audio and video data playback modes;

[0043] The second display module is used to acquire the second audio and video data returned by the target camera device based on the data request command through the audio and video connection, and display the second audio and video data on the second page corresponding to the second mode.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-mentioned audio and video data mode switching method.

[0045] Fourthly, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps performed by the above-described audio and video data mode switching method.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps performed by the above-described audio and video data mode switching method.

[0047] Sixthly, this application also provides a chip, including a processor and an interface for calling and running a computer program, wherein the computer program, when executed by the processor, implements the steps performed by the above-described audio and video data mode switching method.

[0048] The aforementioned audio and video data mode switching method, system, computer device, and computer-readable storage medium, in response to an access request to a target camera device, acquire an audio and video connection, wherein the audio and video connection is a connection between the terminal device and the target camera device used for transmitting audio and video data, and the access request is used to request playback of first audio and video data in a first mode; based on the audio and video connection, display the first audio and video data on a first page corresponding to the first mode; in response to a switching request, determine a second mode corresponding to the switching request, and while maintaining the audio and video connection, send a data request instruction corresponding to the second mode to the target camera device through the audio and video connection, wherein the first mode and the second mode are different audio and video data playback modes; acquire the second audio and video data returned by the target camera device based on the data request instruction through the audio and video connection, and display the second audio and video data on a second page corresponding to the second mode.

[0049] Therefore, during the transition from the first page to the second page, the audio and video connection is maintained and not disconnected. In the second mode, the acquisition of audio and video data no longer depends on the newly established connection. Instead, the target camera changes the source of its audio and video data by issuing a data request command on the existing audio and video connection. Since the audio and video connection is not disconnected, processes such as device wake-up, session establishment, transmission channel negotiation, and waiting for the first frame data, which are only required when the connection is established, are omitted during the switching process. The interaction required between the terminal device and the target camera during the switching process is reduced to a single command issuance and response, thus significantly shortening the switching time. At the same time, since the audio and video connection continues to exist throughout the switching process, the output audio and video data smoothly transitions from the data in the first mode to the data in the second mode, and the picture no longer experiences interruptions due to waiting for the connection to be established, effectively improving the switching efficiency of audio and video data. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an application environment diagram of an audio / video data mode switching method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a method for switching audio and video data modes in one embodiment;

[0053] Figure 3 This is a schematic diagram of the process of obtaining audio and video connections in one embodiment;

[0054] Figure 4 This is a schematic diagram of the process of displaying first audio and video data on a first page in one embodiment;

[0055] Figure 5 This is a schematic diagram of the delayed destruction process of the connection management unit in one embodiment;

[0056] Figure 6 This is a schematic diagram of the mode switching process in one embodiment;

[0057] Figure 7 This is a schematic diagram of the process of displaying second audio and video data on a second page in one embodiment;

[0058] Figure 8 This is a schematic diagram of the structure of the management unit pool in one embodiment;

[0059] Figure 9 This is a schematic diagram of the structure of an audio / video data mode switching system in one embodiment;

[0060] Figure 10 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0063] In related audio and video data mode switching technologies, live streaming and memory card playback correspond to independent display pages. Audio and video connections are established when each display page loads and released when it is destroyed. The existence of audio and video connections is entirely dependent on the existence of the display page. Therefore, page changes inevitably cause audio and video connections to be broken, and the audio and video data required by the new page must rely on a newly established connection to be obtained. This means that each mode switch must go through a complete connection establishment process. The connection establishment process includes multiple stages such as device wake-up, session establishment, transmission channel negotiation, and waiting for the first frame data. Each stage requires round-trip interaction between the terminal device and the target camera device, which takes a long time. Before the process is completed, the terminal device has no audio and video data to display, the picture is interrupted, and the switching efficiency of audio and video data is low.

[0064] For example, in a home security scenario, after receiving an event notification from the door camera, the user opens the application on their terminal device, first viewing the current view of the door on the live stream page, then clicking the playback control to view the recording from ten minutes ago, and finally returning to the live stream page to check the latest situation at the door. In this process, the user triggers two mode switches. In related technologies, each switch requires disconnecting the existing audio / video connection and then re-establishing it. The cumulative waiting time for these two switches may exceed the actual viewing time. Specifically, in this scenario, the user is always viewing the audio / video data from the same target camera. The communication link between the user and the target camera remains unchanged before and after the switch. However, because the audio / video connection depends on the display page, this link is repeatedly disconnected and rebuilt, and the time required for switching cannot be reduced.

[0065] If the above problems are not resolved, users will be continuously affected by video interruptions when switching between the two modes. In monitoring scenarios, they may miss important footage due to waiting. At the same time, the repeated establishment and disconnection of audio and video connections will continue to generate interactions with the server and the target camera device. Under poor network conditions, there is a possibility of failure in establishing a connection each time. The more times the connection is switched, the higher the probability of the screen not displaying properly.

[0066] To address the aforementioned problems, embodiments of this application provide a method for switching audio and video data modes, which can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with target camera device 104 via a network, and terminal 102 and target camera device 104 can also complete device discovery, authentication and signaling relay through server 106.

[0067] The terminal 102 can be, but is not limited to, various smartphones, tablets, personal computers, and portable wearable devices, such as smartwatches, smart bracelets, and head-mounted devices. The terminal 102 runs an application for viewing the images captured by the target camera device 104.

[0068] The target camera device 104 can be, but is not limited to, a home security camera, doorbell camera, baby care camera, commercial surveillance camera, or other IoT device with audio and video acquisition capabilities. The target camera device 104 can be equipped with local storage media such as a memory card to store the audio and video data it acquires; it can also be equipped with a microphone and speaker to acquire ambient sound and play the voice sent by the terminal 102.

[0069] Server 106 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0070] It should be noted that the connection between the terminal 102 and the target camera device 104 for transmitting audio and video data can be a point-to-point connection, a real-time communication connection, or a connection relayed through the server 106. The specific implementation method is not limited in this application embodiment.

[0071] This application provides a method for switching audio and video data modes, such as... Figure 2 As shown, this method is applied to Figure 1 Taking terminal 102 as an example, by maintaining the existing audio / video connection during page changes and requesting audio / video data of another mode on that connection, mode switching no longer depends on the re-establishment of the connection, thereby improving the efficiency of audio / video data switching. The audio / video data mode switching method may include steps 201 to 204, wherein:

[0072] Step 201: In response to the access request to the target camera device, obtain the audio and video connection.

[0073] The audio-visual connection is the connection between the terminal device and the target camera device used to transmit audio and video data, and the access request is used to request the playback of the first audio and video data in the first mode.

[0074] Here, the audio and video connection can be a communication relationship between the terminal device and the target camera device that has completed session establishment and transmission channel negotiation and is capable of carrying audio and video data transmission.

[0075] Understandably, an audio and video connection is not a physical line, but a logical path for audio and video data to be exchanged after a series of interactions between the two parties. Once the path is established, the two parties can send and receive control commands and audio and video streams on it until the path is actively dismantled or disconnected due to an anomaly.

[0076] Playback mode can be used to indicate the source and generation method of audio and video data sent by the target camera device. In an exemplary embodiment, the playback mode may include a live streaming mode and a memory card playback mode. The live streaming mode can be a mode in which the target camera device sends the audio and video data currently acquired and encoded in real time by its audio and video acquisition module to the terminal device, where the time reference of the audio and video data is the current moment. The memory card playback mode can be a mode in which the target camera device reads the stored audio and video data from its memory card according to the time position specified by the terminal device and sends it to the terminal device, where the time reference of the audio and video data is a historical time point specified by the user.

[0077] It should be noted that the two playback modes mentioned above correspond to different display pages in the terminal device's application. This is because the required interface elements and interactive controls differ between the two modes. The page corresponding to the live streaming mode usually includes intercom controls, resolution switching controls, etc.; while the page corresponding to the memory card playback mode usually includes timeline controls, playback progress controls, etc.

[0078] The first mode can be the playback mode corresponding to the audio and video data requested to be played in the access request, and the first audio and video data can be the audio and video data sent by the target camera device in the first mode.

[0079] An access request can be a request to view audio and video data of a target camera device in a certain playback mode. In one exemplary embodiment, the access request can be triggered by a user clicking on the entry corresponding to the target camera device in the device list page of the terminal device. In this case, the first mode can be a live streaming mode. In another exemplary embodiment, the access request can also be triggered by a user clicking on an event message pushed by the server to the target camera device. In this case, the access request requests the playback of historical recordings at the time corresponding to the event. The first mode can be a memory card playback mode.

[0080] Therefore, the first mode is not limited to a live streaming mode. Understandably, in this embodiment, the first mode and the second mode are only used to distinguish between two different playback modes before and after mode switching. Their specific values ​​are determined by the access request and the switching request respectively, and the two can be interchanged. For ease of description, the following mainly describes the case where the first mode is a live streaming mode and the second mode is a memory card playback mode, and the opposite case will be described in the corresponding locations.

[0081] Acquiring an audio / video connection may include establishing a new audio / video connection, or it may include identifying an existing connection between the terminal device and the target camera device as the audio / video connection to be used in this instance. Understandably, this step emphasizes that the terminal device obtains a usable audio / video connection after responding to the access request, and whether this connection is newly established or existing depends on whether the terminal device currently maintains a connection with the target camera device.

[0082] As one implementation, upon receiving an access request, the terminal device can first determine whether it already maintains a connection with the target camera device; if not, it executes the connection establishment process; if already maintained, it directly uses the connection. Alternatively, the terminal device can directly execute the connection establishment process upon receiving the access request, with the connection establishment process itself determining whether each establishment step needs to be actually executed.

[0083] Step 202: Based on the audio and video connection, display the first audio and video data on the first page corresponding to the first mode.

[0084] The first page can be the display page corresponding to the first mode in the application of the terminal device. Continuing from step 201, if the first mode is a live streaming mode, the first page can be the live streaming page; if the first mode is a memory card playback mode, the first page can be the playback page.

[0085] Here, displaying the first audio and video data based on the audio and video connection can refer to the terminal device outputting the audio and video data received and decoded via the audio and video connection to the view component in the first page for displaying the screen and presenting it.

[0086] Understandably, a correspondence needs to be established between the audio / video connection and the first page in order for the audio / video data output by the audio / video connection to be sent to the first page. This is because, in this embodiment, the audio / video connection is not established by the first page itself; the first page does not possess information about the audio / video connection when it has finished loading, and therefore needs to associate the two after the page has loaded.

[0087] As one implementation, after the first page is loaded, the terminal device can register the first page as the data receiver of the audio-video connection and associate the view component in the first page with the data output of the audio-video connection. Thereafter, each frame of audio-video data output by the audio-video connection will be sent to the view component for display.

[0088] Step 203: In response to the switching request, determine the second mode corresponding to the switching request, and while maintaining the audio and video connection, send a data request command corresponding to the second mode to the target camera device through the audio and video connection.

[0089] The first mode and the second mode are different audio and video data playback modes.

[0090] A switching request can be a request to switch from the current playback mode to another playback mode. In an exemplary implementation, the switching request can be triggered by a user clicking a corresponding control on a first page corresponding to the first mode, for example, the user clicking a replay control on the live page, or clicking a live control on the replay page.

[0091] The second mode can be the playback mode requested by the switching request, and the second audio and video data can be the audio and video data sent by the target camera device in the second mode.

[0092] One way to determine the second mode is for the terminal device to read the mode identifier carried in the switching request and determine the playback mode indicated by the mode identifier as the second mode. Another way is for the terminal device to determine the second mode based on the correspondence between the control that triggered the switching request and the current playback mode. For example, if the current playback mode is live streaming, the second mode corresponding to the switching request triggered by the playback control can be determined as the memory card playback mode.

[0093] Maintaining the audio / video connection can be used to indicate that the audio / video connection is not disconnected during the response to a switching request, so that the audio / video connection continues to exist before and after the switching.

[0094] Understandably, in related technologies, the first page calls a stop playback interface to release occupied resources when it finishes displaying. This interface terminates the audio / video connection at the same time as stopping audio / video data playback, so the second page needs to rely on a newly established connection to obtain audio / video data. However, in this embodiment, stopping the playback of first-mode audio / video data and terminating the audio / video connection are processed separately. During the switching process, the terminal device stops the output of first-mode audio / video data to the first page, but retains the audio / video connection itself, so that the connection can continue to be used to carry audio / video data in the second mode.

[0095] It's important to note that the reason these two can be handled separately is that once the audio / video connection is established, the source of the audio and video data it carries can be changed via control commands. The target camera can send its currently acquired data or data from its memory card via this connection; both use the same transmission channel, the only difference being the source from which the target camera reads the data. Therefore, changing the mode of audio / video data does not require changing the connection carrying that data.

[0096] A data request instruction can be a control instruction sent by a terminal device to a target camera device, instructing it to send audio and video data in the second mode.

[0097] As one implementation method, in the case of the second mode being the memory card playback mode, the data request instruction can carry the start time point to be played. After receiving the instruction, the target camera device locates the specified time point from its memory card, reads the corresponding audio and video data, and sends it to the terminal device through the original audio and video connection.

[0098] As another implementation, in the case of the second mode being the live streaming mode, the data request instruction can be an instruction to instruct the resumption of sending real-time acquired data. After receiving the instruction, the target camera device stops reading data from the memory card and instead sends its currently acquired and encoded data to the terminal device via the original audio and video connection.

[0099] Furthermore, the data request instruction can also carry parameters such as resolution level, lens number, playback rate, and whether audio is included, so that the target camera device can send the corresponding audio and video data according to the requirements of the terminal device.

[0100] Step 204: Obtain the second audio and video data returned by the target camera device based on the data request command through the audio and video connection, and display the second audio and video data on the second page corresponding to the second mode.

[0101] The second page can be the display page corresponding to the second mode in the application of the terminal device.

[0102] Here, obtaining the second audio and video data through the audio and video connection can be used to indicate that the audio and video data in the second mode and the audio and video data in the first mode are transmitted to the terminal device through the same audio and video connection.

[0103] Understandably, since the audio and video connection is not disconnected throughout the switching process, the target camera device does not need to re-establish a connection with the terminal device after receiving the data request instruction, and the second audio and video data it sends can directly use the original transmission channel; correspondingly, the terminal device does not need to re-initialize the receiving and decoding process, and can start displaying after receiving the first decodeable data frame in the second mode.

[0104] As one implementation, after the second page is loaded, the terminal device can register the second page as the data receiver of the audio and video connection, and associate the view components in the second page with the data output of the audio and video connection, thereby displaying the second audio and video data on the second page.

[0105] In relevant audio and video data mode switching technologies, the audio and video connection exists solely within the display page; changes to the page inevitably lead to the disconnection and reconstruction of the connection. However, this method of binding the connection to the page suffers from low switching efficiency. Each mode switch requires a complete process of device wake-up, session establishment, transmission channel negotiation, and waiting for the first frame of data. The time consumed in each step accumulates, during which the screen is interrupted. Furthermore, the more times the user switches between the two modes, the longer the cumulative waiting time becomes.

[0106] In the above-described embodiment, the limitations are overcome by maintaining the audio and video connection during the switching process and issuing data request commands over that connection. Although the first page and the second page are two independent display pages, the audio and video data displayed on both pages comes from the same target camera device, and the communication link between the second page and the target camera device does not change substantially before and after the switching. By separating this link from the display page and retaining it during the switching process, the audio and video data required by the second page can be obtained with a single command on the original link, and the various steps in the connection establishment process no longer need to be repeated. In addition, since the audio and video connection continues to exist during the switching process, the previously set device states such as resolution level and lens selection are also continued, without the need to query and restore them again after the switching is completed.

[0107] Compared to related technologies where each switch requires reconnection and results in video interruption, this embodiment reduces the interaction required for mode switching to a single control command issuance and response, significantly shortening the switching time. During the switching process, the video smoothly transitions from data in the first mode to data in the second mode, thereby effectively improving the switching efficiency of audio and video data.

[0108] The above embodiments illustrate the overall flow of the audio and video data mode switching method provided in this application. Based on the above overall flow, the specific implementation of each step will be described below. First, the process of obtaining the audio and video connection in step 201 will be described.

[0109] As described in step 201, obtaining an audio / video connection can involve either establishing a new audio / video connection or retrieving an existing one. To distinguish and process these two scenarios separately, this embodiment introduces a connection management unit, which holds the audio / video connection instead of the display page.

[0110] Based on this, in an exemplary embodiment, such as Figure 3 As shown, the process of obtaining an audio / video connection may include steps 301 to 304, wherein:

[0111] Step 301: In response to the access request, obtain the device identifier of the target camera device.

[0112] The device identifier can be information used to uniquely identify a camera device, such as device serial number, device unique number, device media access control address, etc.

[0113] Understandably, the reason for needing to obtain the device identifier is that the other end of the audio / video connection is the camera device rather than the display page. The same camera device can maintain one audio / video connection at any given time to meet the needs of each display page. Using the device identifier as the basis for differentiation can ensure that the connection of the same camera device is correctly used, and also ensure that the connections of different camera devices are independent of each other.

[0114] As one implementation method, the access request can carry the device identifier of the target camera device, which the terminal device can directly read from the access request. For example, when a user clicks on an item in the device list page, the device identifier associated with that item is written into the access request; when a user clicks on an event push message, the device identifier carried in that message is written into the access request.

[0115] Step 302: Check if there is a connection management unit corresponding to the device identifier.

[0116] The connection management unit is used to maintain the audio and video connection corresponding to the device identifier.

[0117] Here, the connection management unit can be a logical unit in the terminal device used to hold and manage an audio / video connection. Understandably, there is a one-to-one relationship between the connection management unit and the audio / video connection. The connection management unit is responsible for establishing, maintaining, and releasing the audio / video connection, and recording its current status. The connection management unit itself exists independently of the display page; it is not created when a display page is loaded, nor is it released when a display page is destroyed.

[0118] It should be noted that the purpose of introducing the connection management unit is to free the audio / video connection from dependence on the display page. In related technologies, the audio / video connection is held by the display page, and the destruction of the page means that the connection no longer has any users. If it is not destroyed at this time, the connection will become an unmanaged resource, so the destruction of the page and the disconnection of the connection usually occur simultaneously. However, after setting up the connection management unit, the destruction of the display page no longer causes the audio / video connection to lose its holder. The connection continues to be held by the connection management unit, and the timing of its destruction can be determined separately.

[0119] In one exemplary embodiment, the connection management unit may record at least one of the following: a device identifier, used to indicate the camera device corresponding to the connection management unit; a state machine, used to record the current connection stage of the audio / video connection; a mode flag, used to indicate the playback mode of the audio / video data currently carried by the audio / video connection; an identifier of the current display page, used to indicate the page currently displaying audio / video data based on the audio / video connection; a connection status flag, used to indicate whether the audio / video connection has been established and whether audio / video data has started to be output; and a pending destruction flag and a delayed destruction timer, used to control the timing of the destruction of the audio / video connection when no display page is used.

[0120] The identifier of the currently displayed page can be recorded using a weak reference. Here, a weak reference represents a reference relationship that does not affect the release of the referenced object. The referenced object can be released normally when all other references are released, at which point the weak reference automatically becomes invalid. Understandably, the connection management unit's lifespan is longer than the displayed page. If a general reference method is used to record the currently displayed page, pages closed by the user will not be released because they are held by the connection management unit. As users repeatedly enter and exit pages, the memory usage of the terminal device will continue to increase. Using a weak reference avoids this situation.

[0121] Furthermore, considering the mutual reference relationship between the connection management unit and its internal state machine—that is, the connection management unit holds the state machine to read and update the connection phase, while the state machine needs to notify the connection management unit when its state changes—in an implementation environment using reference counting for memory management, this mutual reference will prevent the reference counts of both from reaching zero, thus preventing them from being released. To address this, a proxy object holding a weak reference to the connection management unit can be created. The state machine holds this proxy object instead of directly holding the connection management unit. Notifications generated by the state machine are first sent to this proxy object, which then forwards them to the connection management unit pointed to by its weak reference. In this way, the mutual reference relationship between the two is broken, and the connection management unit can be released normally. After its release, the weak reference held by the proxy object automatically becomes invalid, and its forwarding operation is no longer executed because the target is null. As one implementation method, the proxy object can be implemented as an instance of a class that conforms to the state machine notification protocol. This class holds the connection management unit with a weak reference attribute, and forwards calls to this attribute in its implemented protocol methods. No operation is performed when this attribute is null.

[0122] As a query implementation method, the terminal device can maintain a storage structure indexed by the device identifier, store each connection management unit in it, and search in the storage structure by using the device identifier as the index when querying.

[0123] Step 303: In the absence of a connection management unit, create a connection management unit corresponding to the device identifier, and establish an audio and video connection with the target camera device through the connection management unit.

[0124] The absence of a connection management unit corresponding to the device identifier indicates that the terminal device is not currently maintaining an audio / video connection with the target camera device. Understandably, this situation typically occurs when a user first accesses the target camera device, or when the user has previously accessed the target camera device but the corresponding connection management unit has been destroyed.

[0125] As one way to create a connection management unit, you can allocate storage space for the connection management unit, set its device identifier to the device identifier of the target camera device, set its mode flag to idle, set its connection status flag to not established, and create a state machine and set it to the initial state.

[0126] In one exemplary implementation, the process of establishing an audio-visual connection may include: sending a wake-up command to the target camera device when the target camera device is in a sleep state, and waiting for the target camera device to wake up completely; establishing a session between the terminal device and the target camera device; negotiating an audio-visual transmission channel via the session; and determining that the audio-visual connection is ready when the audio-visual transmission channel outputs the first frame of data.

[0127] The "sleep state" indicates that the main control chip and audio / video acquisition module of the target camera device are in a power-off or low-power state, and are not responding to audio / video data requests. It should be noted that not all camera devices have a sleep state. For camera devices continuously powered by a power adapter, their main control chip and audio / video acquisition module remain powered on and can respond to connection requests at any time; the connection establishment process does not include a wake-up step. However, for battery-powered camera devices, to reduce power consumption, the main control chip and audio / video acquisition module are in a sleep state when no events occur, with only the low-power wake-up module remaining active. In this case, a wake-up process is required before connection establishment.

[0128] As one way to determine whether a target camera is in a sleep state, the terminal device can read the power supply type attribute of the target camera and determine that it may be in a sleep state if it is battery powered. As another way, the terminal device can also query the server for the online status of the target camera and determine that it is in a sleep state if it is in a low-power online state.

[0129] As one way to wake up a target camera device, the terminal device can send a wake-up command through the server. The server relays the command through a low-power long connection maintained between itself and the target camera device. The main control chip and audio / video acquisition module of the target camera device are then powered on and report a wake-up completion notification after they are ready.

[0130] A session can be a logical path between a terminal device and a target camera device used to carry control commands and status notifications. As one way to establish a session, the terminal device and the target camera device can each query their own network address translation (NAT) addresses from the server, exchange their address information via the server, and attempt to establish a direct connection path. If the direct connection fails, the server will act as an intermediary to establish a connection path, followed by key negotiation to complete the session establishment.

[0131] Audio and video transmission channels can be pathways carried over a session for transmitting encoded audio and video streams. As one method of negotiating audio and video transmission channels, a terminal device can send a start transmission command to a target camera device via a session. This command carries parameters such as the requested resolution level and whether audio is included. The target camera device, based on this and its own capabilities, determines the actual transmission parameters, including video encoding format, resolution, frame rate, bit rate, audio encoding format, and sampling rate, and returns these parameters to the terminal device. The terminal device then initializes its decoder accordingly. It should be noted that for a target camera device with multiple lenses, multiple audio and video transmission channels can be negotiated separately over the same session, with each channel carrying the audio and video data captured by its corresponding lens.

[0132] The first frame of data can be the first data frame that the terminal device receives and can independently decode. Understandably, not every frame in the audio and video stream can be decoded independently. The terminal device needs to receive a complete, independently decodeable data frame before it can start decoding and displaying. Therefore, it is used as the criterion for determining whether the audio and video connection is ready.

[0133] Furthermore, in an exemplary embodiment, the state machine in the connection management unit can record the stage of the above-mentioned establishment process. The states contained in the state machine can be divided into states of the establishment phase and states of the holding phase. The states of the establishment phase may include the initial state, wake-up state, connection state, session ready state, channel connection state, output state, channel output success state, and all channel output success state; the states of the holding phase may include the live broadcast ready state, replay state, and paused state; in addition, it may also include the connection failure state.

[0134] The transition conditions for each state are as follows: In the initial state, if an audio / video connection is being established and the target camera device is in sleep mode, the system transitions to the wake-up state; otherwise, it directly transitions to the connection state. In the wake-up state, if the target camera device has finished waking up, the system transitions to the connection state. In the connection state, if the session is successfully established, the system transitions to the session ready state. In the session ready state, if transmission channel negotiation begins, the system transitions to the channel connection state. In the channel connection state, if transmission channel negotiation is completed, the system transitions to the output state. In the output state, if the first frame of data is received, the system transitions to the channel output success state. For target camera devices with multiple lenses, after all transmission channels have successfully output images, the system transitions to the all-channel output success state. Subsequently, depending on the data mode carried by the audio / video connection, the system transitions to either the live streaming ready state or the playback state. Furthermore, in any state, if an abnormal disconnection of the audio / video connection is detected, the system transitions to the connection failure state. In the connection failure state, if the establishment process is re-executed, the system transitions back to the connection state.

[0135] It should be noted that the three states in the hold phase share a common characteristic: the audio and video connection session and transmission channel are both established and not yet terminated. The only difference lies in the data content currently carried by the audio and video connection: in the live-ready state, it carries data collected in real time by the target camera device; in the playback state, it carries data read from the target camera device's memory card; and in the paused state, it does not carry audio or video data. Therefore, the three states can transition between each other without going through any state in the establishment phase. Understandably, the mode switching in this embodiment occurs within the hold phase, which is why the switching process does not need to repeat the steps of the establishment phase.

[0136] Step 304: If a connection management unit exists, determine the connection currently maintained by the connection management unit as an audio / video connection.

[0137] Among them, there is a connection management unit corresponding to the device identifier, which can be used to indicate that the terminal device is still maintaining an audio and video connection with the target camera device.

[0138] Understandably, this situation can occur because the connection management unit does not destroy itself immediately when no page is displayed, but rather destroys itself after a period of time. If the user revisits the target camera device during this delay period, the query in this step will be successful. The relevant details will be explained later.

[0139] As one implementation, the terminal device can directly determine the connection currently maintained by the connection management unit as the audio / video connection used for this access, without executing the establishment process described in step 303. Furthermore, it can also detect whether the connection is still available before making the determination, and re-execute the establishment process if it has been abnormally disconnected.

[0140] Through steps 301 to 304, the acquisition of audio and video connections is divided into two cases: establishment and retrieval, and processed separately. Since the connection maintained by the existing connection management unit is directly retrieved, the steps of device wake-up, session establishment, transmission channel negotiation, and waiting for the first frame data are eliminated, and the response time of this access is greatly shortened. Therefore, the image presentation speed can be improved when the user repeatedly accesses the same target camera device in a short period of time.

[0141] The process of obtaining the audio and video connection has been explained above. After obtaining the audio and video connection, the output audio and video data needs to be displayed on the first page. The following explains the process of displaying the first audio and video data on the first page in step 202.

[0142] Unlike related technologies, the audio and video connection in this application embodiment is not established by the first page itself. The first page does not have the information of the audio and video connection when it is loaded. Therefore, it is necessary to first establish the association between the first page and the audio and video connection, and then decide the subsequent processing method based on the current status of the audio and video connection.

[0143] Based on this, in an exemplary embodiment, such as Figure 4 As shown, the process of displaying the first audio and video data on the first page may include steps 401 to 404, wherein:

[0144] Step 401: Determine the first page as the current display page for the audio / video connection.

[0145] The currently displayed page can be the page that is currently displaying audio and video data based on the audio and video connection. The audio and video data output by the audio and video connection and the notification of changes in the connection status are all sent to this page.

[0146] Understandably, in related technologies, audio and video connections are established and held by the display page itself, and the data and notifications it outputs naturally belong to that page without needing to specify a recipient. However, in this embodiment, the audio and video connection is held by the connection management unit and can be used by multiple display pages in succession. Therefore, it is necessary to clarify which page the data and notifications it outputs should be sent to, and the current display page is used to record this information.

[0147] As one implementation, the first page can register with the connection management unit after it loads, and the connection management unit will set the identifier of the currently displayed page it records as the identifier of the first page. Following step 302, this identifier can be recorded using a weak reference.

[0148] Step 402: Send the current connection status of the audio / video connection to the currently displayed page.

[0149] The current connection state can be used to indicate the availability of the audio / video connection at the current moment. In an exemplary implementation, the current connection state may include at least one of the following: whether a session has been established, whether the transmission channel has started outputting audio / video data, the current state of the state machine, and the current value of the mode flag.

[0150] Understandably, the reason for actively sending the current connection status to the currently displayed page is that the audio and video connection may have already been established before the first page loads. For example, when a user re-enters the page during the delayed destruction period, the audio and video connection is already in a ready state; or, when a user returns to the live stream page from the replay page, the audio and video connection has been maintained throughout the replay.

[0151] In the above scenario, all notifications generated during the establishment of the audio / video connection are sent before the first page loads, and the first page cannot receive them. If the first page passively waits for subsequent status change notifications, it will remain on the loading screen until the next status change occurs. However, once the audio / video connection is fully ready and no further status changes occur, the first page will not display any video. Therefore, it is necessary to resend the current connection status when registering the first page as the currently displayed page, so that it can immediately know the actual status of the audio / video connection.

[0152] As one implementation method, after receiving the registration from the first page, the connection management unit can immediately read the connection status flag, the current state of the state machine and the mode flag recorded by it, and organize them into the current connection status and send them to the first page.

[0153] Step 403: If the current connection status indicates that the audio and video connection is ready, establish a first binding relationship between the rendered view of the first page and the first audio and video data output by the audio and video connection.

[0154] The rendered view can be a view component on the first page used to display audio and video footage. The first binding relationship can be used to represent the correspondence where the first audio and video data output by the audio and video connection is sent to the rendered view for display.

[0155] One way to establish the initial binding relationship is to register the rendering view or its associated processing logic as the receiving object of the audio / video connection's data output callback. Each time the audio / video connection decodes and outputs a frame of audio / video data, it sends it to the rendering view through this callback. Another approach is to pass the rendering view's layer handle to the audio / video connection's decoding output module, which then draws the decoded image directly onto that layer. Yet another method is for the audio / video connection to write the decoded audio / video data into a specified buffer, which the rendering view then reads and displays.

[0156] It should be noted that the establishment and dissolution of the binding relationship are both completed locally on the terminal device, without involving interaction with the target camera device, and therefore incur almost no time overhead. Understandably, this is of great significance to the embodiments of this application: during the mode switching process, the screen moves from the first page to the second page, which is essentially a process of dissolving the binding relationship from the rendering view of the first page and then establishing it with the rendering view of the second page. This process does not need to be completed through network interaction, and therefore does not introduce additional latency.

[0157] Furthermore, if the current connection status indicates that the audio / video connection is still being established, the first page can display a loading prompt and wait for subsequent status change notifications before executing this step once the audio / video connection is ready; if the current connection status indicates that the audio / video connection has been disconnected, the first page can trigger the process of re-establishing the connection.

[0158] Step 404: Display the first audio and video data based on the first binding relationship.

[0159] As one implementation method, the terminal device can send the first audio and video data received and decoded via the audio and video connection to the rendering view of the first page according to the first binding relationship, and the rendering view will present it frame by frame, thereby forming a continuous picture on the first page.

[0160] Through steps 401 to 404, the display page and the audio / video connection are dynamically associated after the page loads, without requiring the page to establish the connection itself. Since the first page can immediately know this fact through the supplementary current connection status when the audio / video connection is ready and then immediately establish the binding relationship, the situation where the page continues to wait due to missing previous status notifications can be avoided, further shortening the display time.

[0161] In the above embodiments, the process of displaying the first audio and video data on the first page was described. After this process, the user may exit the first page, at which point the audio and video connection is no longer used by any display page. Unlike the related art where page destruction releases the connection, this application embodiment handles this situation differently. The following describes the processing procedure of the connection management unit after displaying the first audio and video data.

[0162] As described in step 304, when a connection management unit exists, the connection it maintains can be directly accessed. This is possible because the connection management unit does not immediately destroy itself when no page is being displayed. Therefore, in an exemplary embodiment, as... Figure 5 As shown, after the step of displaying the first audio and video data on the first page, the method may further include steps 501 to 505, wherein:

[0163] Step 501: In response to the exit operation of the currently displayed page, the mapping relationship between the currently displayed page and the audio / video connection is terminated.

[0164] The exit operation can be any action that ends the display of the page, such as clicking the back control, returning to the previous page via gesture, or closing the current page.

[0165] As one implementation, the terminal device can remove the identifier of the currently displayed page recorded in the connection management unit after receiving an exit operation. Afterward, audio and video data and status notifications output by the audio / video connection will no longer be sent to any page.

[0166] Furthermore, at least one of the following operations may be performed in this step: closing the functional sub-channels established on the audio / video connection; pausing the data output of the audio / video connection; and releasing the binding relationship between the rendered view and the audio / video data output by the audio / video connection.

[0167] Functional subchannels can be pathways carried over audio / video connections and used to transmit and receive data for specific functions. For example, a monitoring subchannel might transmit ambient audio captured by a target camera to a terminal device, while an intercom subchannel might transmit voice captured by the terminal device to the target camera. It should be noted that the opening and closing of functional subchannels are achieved through session-based control commands. Closing a functional subchannel only stops the transmission and reception of the corresponding audio data; it does not involve dismantling the session itself.

[0168] Pausing the data output of an audio / video connection can be achieved either by the terminal device sending a pause command to the target camera device via a session to stop it from sending audio / video streams, or by the terminal device locally ceasing the decoding and output of the received streams. Understandably, this type of pause only stops the audio / video connection from outputting data; the session and transmission channel remain intact, thus fundamentally different from disconnection. After executing this operation, the state machine can transition to the paused state.

[0169] It should be noted that the above operations differ from the methods used to stop playback in related technologies. In related technologies, unbinding rendering, closing functional sub-channels, stopping transmission channels, and dismantling the session are executed together, resulting in the complete release of the audio and video connection. However, in this embodiment, the above operations are separated and processed separately. Closing functional sub-channels, unbinding rendering, and pausing data output are executed, while the operations of dismantling the session and transmission channels are postponed to subsequent steps based on a determination result. It is this separation that allows the audio and video connection to persist even after the display page is exited.

[0170] Step 502: Set a destruction marker for the connection management unit and start the delayed destruction timer.

[0171] The "to be destroyed" flag indicates that the connection management unit currently has no display pages in use and is awaiting destruction. The delayed destruction timer determines when to actually perform the destruction.

[0172] Understandably, the reason for not immediately destroying the connection management unit when the page is exited is that a user's exit from the page does not necessarily mean they have finished viewing: the user may return immediately after exiting, or may be redirected from the first page to the second page via other pages. If it were destroyed immediately upon page exit, the audio and video connection would need to be re-established in all of these situations, completely eliminating the possibility of connection reuse; however, setting a delay allows the connection to be reused during that delay period.

[0173] As one implementation method, the duration of the delayed destruction timer can be 5 seconds. It should be noted that the selection of this duration needs to strike a balance between two aspects: if the duration is too short, the connection will have been destroyed when the user re-enters the page in a short time, and reuse will not be possible; if the duration is too long, the audio and video connection and related resources of the target camera device will be unnecessarily occupied even if the user has finished viewing, and it will also affect the battery life of battery-powered cameras.

[0174] As another implementation method, the duration of the delayed destruction timer can be any value between 1 second and 30 seconds. It can be a preset fixed value, or it can be configured by the server. It can also be determined according to the power supply type of the target camera device. For example, a longer duration can be used for continuously powered camera devices, and a shorter duration can be used for battery-powered camera devices.

[0175] Furthermore, in this embodiment, the determination of the absence of a displayed page is achieved by actively releasing the corresponding relationship and setting a destruction flag when the page exits, rather than by detecting the reference status of the connection management unit. Understandably, following step 302, the currently displayed page is recorded as a weak reference. This weak reference automatically expires when its target object is released, but this expiration process does not generate a notification, and the terminal device cannot know this fact at the time it occurs. If the determination relies on detecting the reference status, it is necessary to periodically poll to check whether the weak reference has expired. This not only incurs additional overhead, but the timing of the determination also depends on the polling cycle, resulting in a lag. Page exit, however, is a definite and responsive event. Using it as a trigger point, the terminal device can immediately execute the corresponding processing at the moment the event occurs, ensuring a definite timing of the determination without the need for polling.

[0176] Step 503: Maintain the audio and video connection if the marker to be destroyed has not been cleared and the delayed destruction timer has not been reached.

[0177] The phrase "maintain audio and video connection" can be used to indicate that the session and transmission channel of the audio and video connection are not terminated, but are only in a state of not outputting audio and video data, which is to continue the pause state described in step 303.

[0178] Understandably, since the audio and video connection does not output audio and video data during this period, the resulting network traffic consumption and decoding overhead of the terminal device are both low; and since the session is not torn down, once a display page re-accesses the connection management unit, the data output can be restored immediately without having to re-execute the establishment process.

[0179] Furthermore, to reduce the resource consumption of the target camera during this period, the terminal device can also send instructions to the target camera via a session, instructing it to stop the acquisition and encoding of audio and video while maintaining the session.

[0180] Step 504: If the delayed destruction timer expires and the destruction marker is not cleared, destroy the connection management unit and disconnect the audio and video connection.

[0181] The fact that the delay destruction timer has expired and the destruction mark has not been cleared can be used to indicate that no display page has re-accessed the connection management unit during the entire delay period, meaning that the user has indeed ended viewing the target camera device.

[0182] As one implementation, the process of destroying the connection management unit and disconnecting the audio / video connection may include: sending a stop command to the target camera device via the session; disconnecting the transmission channel; dismantling the session; clearing the state machine; and releasing the storage space occupied by the connection management unit. After destruction, if the user accesses the target camera device again, the query in step 302 will no longer be successful, and the establishment process described in step 303 must be re-executed.

[0183] Step 505: If the connection management unit is found before the delayed destruction timer expires, the destruction mark is cleared and the audio / video connection is maintained.

[0184] The fact that the connection management unit was found before the delayed destruction timer expired can be used to indicate that the user revisited the target camera device during the delay period, that is, the query in step 302 hit the connection management unit.

[0185] As one implementation, the terminal device can clear the pending destruction mark of the connection management unit after the query is successful, and the delayed destruction will be cancelled accordingly. The audio and video connection will continue to exist and will be determined as the audio and video connection used for this access as described in step 304.

[0186] Furthermore, after cancellation and destruction, the terminal device may also perform at least one of the following operations: restore the data output of the audio and video connection, causing the state machine to jump from the paused state to the hold phase state corresponding to the mode; compare the first mode requested by this access with the current mode flag of the connection management unit, and if the two are inconsistent, cause the audio and video connection to carry the audio and video data in the first mode according to the mode switching process described below.

[0187] In addition, considering that users may enter and exit the same target camera device's page multiple times in a short period of time, the following operations can be further performed: each time a destruction marker is set, the destruction token of the connection management unit is incremented, and the destruction token at the time the delayed destruction timer is started is recorded; when the delayed destruction timer expires, the recorded destruction token is compared with the current destruction token of the connection management unit, and if the two are consistent, the destruction operation described in step 504 is performed, and if the two are inconsistent, the destruction is not performed.

[0188] The destruction token is used to distinguish the validity of multiple delayed destruction timers when they coexist. Understandably, once a delayed destruction timer is started, the processing to be executed upon its expiration is already in the execution sequence, and this processing is usually irreversible. If the user re-enters the page before the timer expires, the destruction marker is cleared, and then the user exits the page again, a second delayed destruction timer will be started. At this time, the processing corresponding to the first timer still exists, and its expiration time is earlier. In this situation, if the judgment is based solely on the existence of the destruction marker, when the first timer expires, since the user has exited again and the destruction marker has been reset, the judgment is true, and the first timer will execute destruction; however, at this point, there is not enough delay time before the user's second exit, and the connection should remain open. With the introduction of the destruction token, the token recorded in the first timer is inconsistent with the current token, and therefore will not execute destruction; only the last started timer can trigger destruction.

[0189] For example, a user enters and exits the page three times within one second: Upon the first exit, the token is incremented from 0 to 1, timer A is started and the token is recorded as 1; upon re-entry, the token is cleared. Upon the second exit, the token is incremented to 2, timer B is started and the token is recorded as 2; upon re-entry, the token is cleared. Upon the third exit, the token is incremented to 3, timer C is started and the token is recorded as 3. Afterwards, timer A expires, its recorded token is 1 while the current token is 3, so the token is not cleared and destruction is not performed; timer B expires, its recorded token is 2 while the current token is 3, so the token is not cleared and destruction is not performed; timer C expires, its recorded token is 3 and matches the current token, and the token is not cleared, so destruction is performed.

[0190] In addition, for situations where the terminal device is moved to the background, at least one of the following operations can be performed: If the terminal device is moved to the background and the currently displayed page has not performed an exit operation, the above-mentioned delayed destruction process is not triggered, the audio and video connection continues to be maintained, and its data output can be paused to reduce resource consumption, and then resumed when the device is restored to the foreground; If the terminal device is moved to the background and the delayed destruction timer is in progress, and the timer is suspended due to system restrictions on background applications, the processing to be performed after its expiration can be performed after the terminal device is restored to the foreground, and the determination is still made in the above manner; If the terminal device has been in the background for a long time and the audio and video connection has been disconnected, the terminal device obtains the connection status of the audio and video connection when it is restored to the foreground, and re-establishes a connection with the target camera device if the connection is indicated to be disconnected.

[0191] Through steps 501 to 505, the audio and video connection is retained for a period of time after the user exits the displayed page, instead of being immediately released when the page is destroyed. Since the audio and video connection can be directly accessed when the user re-enters the page within this timeframe, the video can be displayed immediately. Furthermore, with the destruction token, the situation of rapidly entering and exiting the page can also be handled correctly, preventing the premature destruction of connections that should still be retained. Therefore, while ensuring rapid connection reuse, long-term resource occupation can be avoided.

[0192] In the above embodiments, the processing procedure of the connection management unit after the display page is exited was described. This process explains why the audio / video connection can continue to exist when no page is in use, while when a page is in use, the audio / video connection needs to switch the data it carries between two playback modes. The mode switching process in step 203 will be described below.

[0193] As described in step 203, the difference between this application embodiment and related technologies lies in separating the processing of stopping the playback of the first mode audio and video data from the process of disconnecting the audio and video connection. Based on this, in an exemplary embodiment, as... Figure 6 As shown, the mode switching process may include steps 601 to 604, wherein:

[0194] Step 601: In response to the switching request, while maintaining the audio and video connection, the first binding relationship between the rendered view of the first page and the first audio and video data output by the audio and video connection is released.

[0195] The option to maintain audio and video connectivity can be used to indicate that the session and transmission channel will not be terminated during this step and subsequent steps.

[0196] One way to unbind the first binding relationship is to set the data output callback of the audio / video connection to null; another way is to set the view handle held by the decoding output module to null or pass in an empty object; yet another way is to disconnect the association between the rendering view and the data buffer. These methods correspond to the methods for establishing the first binding relationship described in step 403.

[0197] Understandably, the reason for needing to unbind the first binding relationship is that the first page may not have been completely destroyed during the switching process; if it is not unbound, the audio and video data output in the second mode by the subsequent audio and video connection may be sent to the rendering view of the first page, resulting in abnormal screen display.

[0198] Furthermore, this step can also disable functional sub-channels established over the audio / video connection. For example, when switching from live streaming mode to memory card playback mode, the monitoring sub-channel and intercom sub-channel can be disabled: after the switch, the audio played comes from the recording on the memory card instead of real-time capture, and the transmission of real-time ambient audio is no longer needed; the user is viewing historical recordings, and the intercom function is no longer applicable. Following step 501, disabling the functional sub-channel is achieved through session-based control commands and does not involve dismantling the session itself. Additionally, other supplementary functions applicable only to the first mode can also be disabled, such as image enhancement functions used to process real-time captured images.

[0199] It should be noted that disabling a function subchannel is not a necessary operation for mode switching. If a function subchannel is needed in both the first and second modes, it can remain open; for example, when switching from memory card playback mode to live streaming mode, if the user wants to hear ambient sounds immediately after the switch, they can either retain or re-enable the monitoring subchannel.

[0200] In addition, to make the screen transition during the switching process more natural, when the first binding relationship is released, the display of the last frame image in the first page rendering view can be retained until the second page rendering view begins to display the second audio and video data.

[0201] Step 602: Change the current display page of the audio / video connection from the first page to the second page.

[0202] In one implementation, the first page can initiate a deregistration with the connection management unit after triggering a switching request. The connection management unit then removes the identifier of the currently displayed page it has recorded. Subsequently, the terminal device loads the second page, and after loading, the second page initiates a registration with the connection management unit. The connection management unit then sets the identifier of the currently displayed page it has recorded as the identifier of the second page. After this step, the audio and video data and status notifications output by the audio and video connection are then sent to the second page.

[0203] It should be noted that this step differs from the step 501 in that it removes the corresponding relationship: in step 501, after the identifier of the currently displayed page is removed, no new value is set, and the connection management unit enters a state awaiting destruction; however, in this step, after the identifier is removed, it is immediately set to the identifier of the second page, and the connection management unit does not enter a state awaiting destruction. Therefore, the mode switching process does not trigger the delayed destruction process.

[0204] Furthermore, if there is a time interval between the cancellation of the first page and the registration of the second page, the identifier of the currently displayed page will be empty during this interval, and the data output by the audio / video connection will not have a receiver. This can be handled in any of the following ways: pause the data output of the audio / video connection during this interval and resume it after the second page is registered; or discard the data output by the audio / video connection during this interval; or combine the cancellation of the first page and the registration of the second page into one process, so that the identifier of the currently displayed page changes directly from the first page to the second page without going through an intermediate state of being empty.

[0205] Step 603: Update the mode marker of the audio / video connection from the first mode to the second mode.

[0206] The mode marker can be used to indicate the playback mode of the audio and video data currently carried by the audio and video connection.

[0207] Understandably, the reason for recording and updating the mode marker is that the existence of the audio / video connection is independent of the displayed page, and the mode of the data it carries needs to be persistently recorded for later querying. For example, when querying the connection management unit for the current connection status after the second page loads, it needs to determine whether the audio / video connection is currently carrying data in the second mode based on the mode marker, thus deciding whether to wait or display directly; when the user triggers a switching request again, it needs to determine the current playback mode based on the mode marker, thus determining the mode to switch to; when the user re-enters the page during the delayed destruction period, it needs to determine whether the mode requested by the user is consistent with the mode currently carried by the audio / video connection based on the mode marker. If the mode marker is not recorded, all of the above points would need to query the target camera device for the current status, resulting in additional interactions.

[0208] In one implementation, the mode flag can be updated before sending the data request command, in which case it reflects the mode desired by the terminal device. Alternatively, the mode flag can be updated after receiving a response from the target camera device, in which case it reflects the mode actually carried by the audio / video connection. Furthermore, both the desired mode and the actual mode can be recorded simultaneously to distinguish between in-process and completed handover during the handover process.

[0209] Step 604: Send a data request command corresponding to the second mode to the target camera device via audio / video connection.

[0210] As one implementation method, the data request command can be sent via a session connected to the audio / video connection. Continuing from step 203, when the second mode is the memory card playback mode, the data request command can carry parameters such as the start time of playback, shot number, playback rate, and whether audio is included; when the second mode is the live streaming mode, the data request command can carry parameters such as resolution level, shot number, and whether audio is included.

[0211] It should be noted that the data request instruction sent in this step differs from the start transmission instruction described in step 303. The latter is used to negotiate and establish an audio / video transmission channel, and it is sent before the transmission channel exists. The former, however, is sent when the transmission channel already exists, and its function is only to change the source of the data carried by that transmission channel. Therefore, this step does not involve renegotiating the transmission channel.

[0212] Furthermore, if the audio and video data in the second mode differs from those in the first mode in terms of encoding parameters, such as different resolutions or frame rates, the target camera device can insert an updated parameter set into the bitstream when switching data sources. The terminal device's decoder then reconfigures the parameters required for decoding based on this parameter set and continues decoding. Understandably, the above process is completed within the transmission channel, without the need to renegotiate the transmission channel or dismantle and rebuild the audio and video connection.

[0213] After the target camera device completes the data source switch, the state machine can transition from the hold phase state corresponding to the first mode to the hold phase state corresponding to the second mode, for example, from the live broadcast ready state to the playback state, or from the playback state to the live broadcast ready state. Following step 303, the above transition process does not involve any state in the setup phase.

[0214] Furthermore, the above steps also apply to the scenario of switching from memory card playback mode to live streaming mode. In this case, the first mode is memory card playback mode, the second mode is live streaming mode, the first page is the playback page, and the second page is the live streaming page. As one implementation, the switching process in this scenario may include: responding to a user's return or real-time operation on the playback page, determining the second mode as live streaming mode; sending a stop playback command to the target camera device via the audio / video connection, instructing it to stop reading data from the memory card; releasing the binding relationship between the rendered view of the playback page and the audio / video data output by the audio / video connection, and releasing playback-related local resources; changing the identifier of the currently displayed page from playback page to live streaming page; updating the mode flag from memory card playback mode to live streaming mode; and sending a resume live streaming command to the target camera device via the audio / video connection.

[0215] It should be noted that both the stop playback command and the resume live broadcast command are issued via the session, and their execution does not involve the termination of the session and transmission channel; in one exemplary implementation, the two can also be combined into a single command. Since the switching processes in both directions are symmetrical, users can switch back and forth between the two modes without needing to re-establish the audio and video connection for each switch.

[0216] In addition, to ensure that the functionality remains available under various abnormal conditions, at least one of the following operations can be performed:

[0217] In response to a switching request, the connection status of the audio / video connection is obtained. If the connection status indicates that the audio / video connection has been disconnected, a new connection is established with the target camera device, and the second audio / video data is obtained through the newly established connection.

[0218] Understandably, audio and video connections may be interrupted due to network outages, offline target camera devices, or long background processes on terminal devices. In such cases, the premises upon which the embodiments of this application rely no longer hold, and the connection can be re-established using the methods employed in related technologies. In this situation, the user may perceive a brief loading process, but the functionality remains available.

[0219] One way to determine whether an audio / video connection has been disconnected is to read the connection status flag of the connection management unit; another way is to check whether the session keep-alive mechanism has received a response within a preset time period; yet another way is to check whether the transmission channel has received data within a preset time period.

[0220] If no second audio or video data is received from the target camera within a preset time after sending the data request command, a loading timeout message will be displayed on the second page, and a retry entry will be provided. After the user triggers a retry, the data request command will be resent. If the retry is still unsuccessful, the user can further revert to the method of re-establishing a connection.

[0221] If a new switching request is received after a data request command is sent but before a response is received from the target camera device, the new switching request is ignored. Understandably, there is a time interval between the issuance of the data request command and the response from the target camera device. If a new command is issued again within this interval, the target camera device may receive a disordered sequence of commands, resulting in the data it sends being inconsistent with the pattern expected by the terminal device. For example, if a user quickly switches from live to replay and then back to live, and the interval between the two commands is less than the round-trip time, the target camera device may receive the replay command and the live command sequentially, and the order of its responses may be reversed due to different processing times, ultimately causing the terminal device to receive replay data when it is on the live page.

[0222] One implementation approach is to set a flag indicating that a handover is in progress in the connection management unit. When a data request command is issued, the flag is set to "established," and when a response is received or a timeout occurs, the flag is set to "disestablished." Handover requests received during the "established" period are ignored. Another implementation approach is to assign a sequence number to each data request command and process only the response corresponding to the command with the largest sequence number.

[0223] If the audio / video connection is detected to be disconnected during the delayed destruction timer, the connection management unit is still destroyed as described in step 504, so that subsequent accesses will re-execute the establishment process.

[0224] Through steps 601 to 604, the mode switching is implemented as a transfer of the binding relationship, a change of the currently displayed page, an update of the mode flag, and the issuance of a data request command. None of these operations involve the teardown of the session or transmission channel. Since the various steps in the connection establishment process no longer need to be executed during the switching process, the interaction required for the switching process is reduced to the issuance and response of a single control command. Therefore, the time required for mode switching can be significantly shortened. At the same time, in conjunction with the above-mentioned exception handling operations, the method automatically reverts to re-establishing the connection in cases such as when the audio or video connection has been broken. Therefore, while improving the switching efficiency under normal circumstances, the availability under abnormal circumstances is not reduced.

[0225] In the above embodiments, the mode switching process has been described. After the switching is completed, the audio and video connection carries audio and video data in the second mode, which needs to be displayed on the second page. The process of displaying the second audio and video data on the second page in step 204 will be described below.

[0226] Similar to the process of displaying the first audio / video data on the first page in step 202, the second page is also not the initiator of the audio / video connection. Therefore, it also needs to first know the current status of the audio / video connection before establishing the binding relationship. Based on this, in an exemplary embodiment, such as... Figure 7 As shown, the process of displaying the second audio and video data on the second page may include steps 701 to 703, wherein:

[0227] Step 701: Send the current connection status of the audio / video connection to the second page.

[0228] As one implementation method, after registering the second page as the current display page, the connection management unit can immediately read the connection status flag, the current state of the state machine and the mode flag it records, and organize them into the current connection status and send them to the second page.

[0229] It's important to note that the situation on the second page differs from that on the first page. When the first page loads, the audio / video connection may still be being established or may already be ready; however, when the second page loads, the audio / video connection is usually already established, and the connection is either in the process of mode switching or has already been completed. Therefore, the current connection status sent to the second page can include the current value of the mode flag, allowing the second page to determine whether the audio / video connection is currently carrying data in the second mode.

[0230] As one implementation, the second page can determine the subsequent processing method based on the received current connection status: if the current connection status indicates that the audio / video connection is ready and the mode marker is already in the second mode, steps 702 and 703 are executed; if the current connection status indicates that the audio / video connection is ready but the mode marker is still in the first mode, it indicates that the mode switching has not yet been completed, and the second page can display a loading prompt and wait for the notification that the switching is complete; if the current connection status indicates that the audio / video connection is not ready or has been disconnected, it can be handled according to the exception handling method described above.

[0231] Step 702: If the current connection status indicates that the audio and video connection is ready, establish a second binding relationship between the rendered view of the second page and the second audio and video data output by the audio and video connection.

[0232] The second binding relationship can be used to represent the correspondence between the second audio and video data output by the audio and video connection and the rendering view of the second page for display. The implementation method for establishing the second binding relationship is the same as that for establishing the first binding relationship described in step 403, and can be achieved by callback registration, view handle passing, or data buffer sharing.

[0233] Step 703: Display the second audio and video data based on the second binding relationship.

[0234] As one implementation method, the terminal device can send the second audio and video data, received and decoded via the audio and video connection, to the rendering view of the second page according to the second binding relationship, and the rendering view will then present it frame by frame. At this point, the mode switching process is complete, and what the user perceives is that the screen of the first page is replaced by the screen of the second page, without any black screen or loading prompts appearing during this process.

[0235] In addition, when the second mode is the memory card playback mode, the second page usually needs to display playback event data in addition to the second audio and video data. The playback event data can be used to present the user with the time intervals or event entries corresponding to the data stored on the target camera's memory card, allowing the user to select the desired playback position.

[0236] Understandably, playback event data needs to be queried from the target camera device. If the query is performed after the second page has loaded, the timeline control or event list on the second page will be blank until the query is completed. At this time, although the user can see the audio and video, they cannot select other playback positions. To address this, the following operations can be performed: while the first audio and video data is displayed on the first page, the playback event data from the target camera device is retrieved and written to the local cache; in response to a switching request, the playback event data is read from the local cache and displayed on the second page, and updated playback event data is requested from the target camera device; if the updated data is obtained, the playback event data displayed on the second page and the local cache are updated.

[0237] The process of acquiring playback event data during the display of audio and video data on the first page can be termed preloading. As an implementation method, to avoid the bandwidth consumed by preloading affecting the transmission of audio and video data in the first mode, preloading can be performed after a delay following the display of the first frame of the first page, or when the current network condition is detected to be good.

[0238] As one way to write to the local cache, the acquired playback event data can be merged first: adjacent event entries with a time interval less than a preset deduplication time are merged into a single entry and then written to the cache. The preset deduplication time can be 5 minutes. It should be noted that the reason for merging is that when the target camera device is continuously active in the same scene, it may generate multiple event records consecutively. If each record is presented, the event list will be too long and not conducive to user search.

[0239] As one way to manage local caching, the playback event data stored in the local cache can have a preset caching time window. Data exceeding this window is cleared, and the caching time window can be 3 days. It should be noted that the reason for setting a caching time window is that the memory card capacity of the target camera device is limited, and older data may have been overwritten in cycles, causing the corresponding records in the local cache to become invalid. At the same time, limiting the size of the cache can also prevent the continuous growth of the terminal device's storage space.

[0240] As one way to obtain updated data, for target cameras that support incremental queries, only data added after the latest record in the local cache can be requested. This method requires less data transmission and has a faster response. For target cameras that do not support incremental queries, all data within a specified time range can be requested, and the obtained data can replace the content of the corresponding time range in the local cache. As one way to determine whether a target camera supports incremental queries, the capability information reported by the target camera can be read. Another way is to determine this based on the firmware version of the target camera. Yet another way is to first attempt an incremental query, and if the target camera returns a response indicating that it does not support incremental queries, then request all data instead.

[0241] It should be noted that the above-mentioned preloading and caching processes are independent of the mode switching process. They can be implemented separately or in combination. When implemented in combination, the audio and video images and playback event data can be presented immediately after the user switches to the second page.

[0242] Through steps 701 to 703, the second page also learns the status of the audio and video connection through the received current connection status and immediately establishes a binding relationship, so that the screen can be displayed immediately after the switch is completed; with the preloading and caching of the playback event data mentioned above, the auxiliary information in the second page can also be displayed without waiting for the query to be completed, so the display speed of the second page content after the mode switch can be further improved.

[0243] The above embodiments are all illustrated using the scenario of a terminal device accessing a single target camera device. In actual use, users may view multiple camera devices simultaneously or switch between multiple camera devices. The following describes the scenario where a terminal device simultaneously manages audio and video connections to multiple camera devices.

[0244] As described in step 302, there is a one-to-one correspondence between the connection management unit and the device identifier. This correspondence allows the connection management units of multiple camera devices to be stored and queried separately according to the device identifier. Based on this, in an exemplary embodiment, the target camera device is the first camera device among multiple camera devices, and the multiple camera devices also include a second camera device, such as... Figure 8 As shown, the method may further include steps 801 to 802, wherein:

[0245] Step 801: Store the connection management units corresponding to each of the multiple camera devices into the connection management unit pool according to the device identifier of each camera device.

[0246] Each connection management unit maintains an audio and video connection with its corresponding camera device.

[0247] Here, the connection management unit pool can be a container in the terminal device used to store multiple connection management units, which organizes the connection management units using the device identifier as an index. As one implementation, the connection management unit pool can be implemented using a mapping structure with the device identifier as the key and the connection management unit as the value.

[0248] Understandably, after the introduction of the connection management unit pool, the operations involving the connection management unit in the aforementioned embodiments are adjusted accordingly: the query in step 302 is to query the connection management unit pool using the device identifier as the index; the connection management unit created in step 303 is added to the connection management unit pool after creation; the connection management unit destroyed in step 504 is removed from the connection management unit pool before destruction.

[0249] Furthermore, the connection management unit pool can also have a capacity limit. If the number of connection management units in the pool reaches the limit and a new one needs to be created, a connection management unit can be selected and destroyed based on the principle of the least used unit, thus freeing up space for the new connection management unit. As one implementation method, this capacity limit can be determined based on the processing power of the terminal device and the network bandwidth.

[0250] Step 802: If the audio / video connection corresponding to the first camera device is switched from the first mode to the second mode, maintain the audio / video connection corresponding to the second camera device.

[0251] Understandably, each connection management unit is distinguished from the others by device identifier. Each connection management unit maintains an audio / video connection with its corresponding camera device and has its own state machine, mode flag, current display page identifier, connection status flag, pending destruction flag, and destruction token. The sessions and transmission channels of each audio / video connection are also established, maintained, and terminated separately. Therefore, mode switching, page exit, and connection destruction performed on the first camera device do not affect the audio / video connection corresponding to the second camera device.

[0252] For example, when a user switches from live streaming to playback while viewing the footage from the first camera device, the audio and video connection of the second camera device remains unaffected, and its mode markers, state machine, and delayed destruction timer remain unchanged. When the user then switches to view the second camera device, if its connection management unit is still within the delayed destruction period, the connection it maintains can be directly accessed as described in step 304.

[0253] As one application, the aforementioned multi-device scenario can be applied to multi-view preview scenarios. The terminal device simultaneously displays feeds from multiple cameras on the same page. Each feed corresponds to a connection management unit, and the rendered view of each feed is bound to the audio / video connection maintained by its respective connection management unit. The user can independently switch modes for any one feed without affecting the others. As another application, it can also be used for scenarios involving back-and-forth viewing between devices: since each connection management unit remains active for a period after the page is closed, the connection can be reused when the user returns to a previously viewed camera within a short time.

[0254] It should be noted that the multiple camera devices mentioned in this embodiment are different from the multiple transmission channels of the multi-lens camera device mentioned above. The former refers to multiple physically independent camera devices, each with its own device identifier and independent audio and video connection; the latter refers to multiple lenses on the same camera device, whose corresponding multiple transmission channels are carried on the same session of the same audio and video connection.

[0255] Through steps 801 to 802, the audio and video connections of multiple camera devices are isolated and managed independently according to device identification. Since the operation performed on any one camera device is limited to its own connection management unit, the connections and status of other camera devices are not affected. Therefore, while supporting users to view multiple camera devices simultaneously, it is ensured that the mode switching of each device does not interfere with each other.

[0256] The above embodiments describe the specific implementation of each step in the audio and video data mode switching method provided in this application. Below, a complete usage process will be used to explain the cooperative relationship between the above embodiments. In this usage process, an example is a doorbell camera powered by a battery, with a 5-second delay timer for data destruction.

[0257] When a user clicks on the doorbell camera entry in the device list page of a terminal device, the terminal device generates an access request carrying the device identifier of the camera. The first requested mode is live streaming mode. The terminal device obtains the device identifier from the access request and queries the connection management unit pool using that identifier. If no corresponding connection management unit is found, a connection management unit is created and added to the connection management unit pool, its mode flag is set to idle, and the state machine is set to the initial state. Subsequently, the terminal device receives a wake-up command from the server, and the state machine transitions to the wake-up state. After the camera is woken up, the terminal device begins to establish a session, and the state machine transitions to the connection state. After the session is established, it transitions to the session ready state. Then, it negotiates the audio and video transmission channels, transitioning sequentially to the channel connection state and the output state. After receiving the first frame of data, it transitions to the channel output success state and the all-channel output success state, the mode flag is set to live streaming mode, and the state machine transitions to the live streaming ready state.

[0258] Simultaneously, the live streaming page loads and registers as the currently displayed page with the connection management unit. The connection management unit sends the current connection status to it. Since the audio and video connection is still being established, the live streaming page displays a loading prompt. Once the audio and video connection is ready, the live streaming page receives the status change notification, establishes the first binding relationship between its rendered view and the audio and video data output by the audio and video connection, and the picture begins to be displayed. The user enables the listening function on the live streaming page and sets the resolution to high definition. These settings are sent to the camera via the session. During the picture display, the terminal device queries the camera for the playback event data of the day and writes it to the local cache.

[0259] Subsequently, the user clicks the playback control to view the recording from ten minutes prior. The terminal device generates a switch request and determines that the second mode is the memory card playback mode. While maintaining the audio and video connection, it sends a command to disable listening and a command to disable intercom via the session, thus breaking the first binding relationship between the live stream page rendering view and the audio and video data. It loads the playback page and changes the currently displayed page's identifier from the live stream page to the playback page; it updates the mode marker from real-time live stream mode to memory card playback mode; and it sends a data request command via the audio and video connection, carrying the time point selected by the user. After the playback page loads, it reads playback event data from the local cache and displays it in the timeline control, allowing the user to see the distribution of the recorded segments for the day. Simultaneously, the terminal device initiates an incremental query to the camera to obtain the latest data. After receiving the data request instruction, the camera locates the specified time point from its memory card, reads the audio and video data, and sends it to the terminal device through the original transmission channel. The state machine jumps from the live broadcast ready state to the playback state. The playback page establishes a second binding relationship between its rendered view and the audio and video data output by the audio and video connection, and the screen changes from the live broadcast screen to the playback screen without any black screen.

[0260] After the user reviews the recording, they click the return control. The terminal device generates a switch request and confirms that the second mode is live streaming mode. It then issues a stop playback command via the session, unbinds the playback page from its binding, releases playback-related local resources, changes the currently displayed page's identifier back to the live streaming page, updates the mode marker to live streaming mode, and issues a resume live streaming command via the session. The camera then resumes transmitting real-time data, and the state machine transitions from playback state to live streaming ready state. Because the session is not terminated throughout the process, the previously set high-definition resolution remains effective and does not need to be reset.

[0261] Subsequently, the user clicks "back" to exit to the device list page. The terminal device removes the identifier of the currently displayed page recorded by the connection management unit, issues a "close listening" command via the session, pauses the data output of the audio and video connection, and jumps the state machine to the paused state. It sets a "to be destroyed" flag for the connection management unit, increments the destruction token from 0 to 1, starts the first delayed destruction timer, and records its token as 1. At this time, neither the session nor the transmission channel has been terminated. After the user stays on the device list page for about 2 seconds, they click on the camera's entry again. The terminal device queries the connection management unit pool, clears its "to be destroyed" flag, identifies the currently maintained connection as the audio and video connection for this access, and resumes data output. The state machine jumps from the paused state back to the live streaming ready state. After the live streaming page loads, it is registered as the currently displayed page. The connection management unit sends the current connection status to it. Since this status indicates that the audio and video connection is ready, the live streaming page directly establishes the binding relationship and displays the screen without any loading prompt.

[0262] After the user finished viewing and exited, the terminal device reset the destruction token, incrementing it to 2, initiating a second delayed destruction timer, and recording the token as 2. The user did not subsequently access the camera's page. When the first timer expired, the recorded token was 1, while the current token was 2; since they were inconsistent, destruction was not performed. When the second timer expired, the recorded token was 2 and matched the current token; the destruction token was not cleared, so destruction was performed: a stop command was issued, the transmission channel was dismantled, the session was terminated, the state machine was cleared, the connection management unit was removed from the connection management unit pool, and its storage space was released.

[0263] During the complete usage process described above, the user triggered two mode switches and one page re-entry. In related technologies, the audio and video connections need to be re-established at all three points. However, in this embodiment, the entire process only involves establishing a connection once initially, the two mode switches each require only one control command issuance and response, and the page re-entry directly utilizes the existing connection. Therefore, the efficiency of audio and video data switching and the speed of image presentation can be significantly improved.

[0264] In another exemplary embodiment, the first mode can also be a memory card playback mode. For example, after the doorbell camera detects someone passing by, it records a video and pushes an event message to the terminal device via the server. When the user clicks on the message, the terminal device generates an access request, requesting the memory card playback mode as the first mode and carrying the time point corresponding to the event. The terminal device queries the connection management unit pool but fails to find a match, so it creates a connection management unit and establishes an audio-visual connection. After the transmission channel is established, it issues a data request instruction for playback mode, the mode flag is set to memory card playback mode, the state machine jumps to the playback state, and the playback page displays the recording corresponding to the event. After the user finishes viewing, they click the real-time control to confirm the current situation at the door. At this time, the second mode is the live streaming mode, and the terminal device performs the switching according to the aforementioned mode switching process, and the audio-visual connection remains unbroken. Thus, it can be seen that the audio-visual connection established for the first time in this embodiment can be used to carry data in both live streaming mode and memory card playback mode, and the values ​​of the first mode and the second mode can be interchanged.

[0265] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0266] In other embodiments, this application proposes an audio / video data mode switching system applied to a terminal device communicating with a target camera device. Please refer to [reference needed]. Figure 9 The audio and video data mode switching system includes:

[0267] The connection module is used to obtain an audio-visual connection in response to an access request to the target camera device; wherein, the audio-visual connection is a connection between the terminal device and the target camera device for transmitting audio-visual data, and the access request is used to request the playback of the first audio-visual data in the first mode;

[0268] The first display module is used to display first audio and video data on a first page corresponding to the first mode based on the audio and video connection.

[0269] The switching module is used to respond to a switching request, determine the second mode corresponding to the switching request, and send a data request command corresponding to the second mode to the target camera device through the audio and video connection while maintaining the audio and video connection; wherein, the first mode and the second mode are different audio and video data playback modes;

[0270] The second display module is used to acquire the second audio and video data returned by the target camera device based on the data request command through the audio and video connection, and display the second audio and video data on the second page corresponding to the second mode.

[0271] In one exemplary embodiment, the connection module includes an identifier acquisition unit, a query unit, a connection establishment unit, and a connection multiplexing unit, wherein:

[0272] The identifier acquisition unit is used to acquire the device identifier of the target camera device in response to an access request;

[0273] The query unit is used to query whether a connection management unit corresponding to the device identifier exists; wherein, the connection management unit is used to maintain the audio and video connection corresponding to the device identifier;

[0274] The connection establishment unit is used to create a connection management unit corresponding to the device identifier when no connection management unit exists, and to establish an audio and video connection with the target camera device through the connection management unit;

[0275] The connection multiplexing unit is used to identify the connection currently maintained by the connection management unit as an audio / video connection when a connection management unit is present.

[0276] In one exemplary embodiment, the connection establishment unit is further configured to: send a wake-up command to the target camera device when the target camera device is in a sleep state, and wait for the target camera device to wake up completely; establish a session between the terminal device and the target camera device; negotiate an audio / video transmission channel via the session; and determine that the audio / video connection is ready when the audio / video transmission channel outputs the first frame of data.

[0277] In one exemplary embodiment, the first display module includes a page determination unit, a status sending unit, a first binding unit, and a first display unit, wherein:

[0278] The page determination unit is used to determine the first page as the current display page of the audio / video connection;

[0279] The status sending unit is used to send the current connection status of the audio and video connection to the currently displayed page;

[0280] The first binding unit is used to establish a first binding relationship between the rendered view of the first page and the first audio and video data output by the audio and video connection when the current connection status indicates that the audio and video connection is ready.

[0281] The first display unit is used to display the first audio and video data based on the first binding relationship.

[0282] In one exemplary embodiment, the audio / video data mode switching system further includes a destruction module, wherein:

[0283] The destruction module is used to respond to the exit operation of the currently displayed page, to disconnect the correspondence between the currently displayed page and the audio / video connection; to set a destruction mark for the connection management unit and start a delayed destruction timer; to maintain the audio / video connection if the destruction mark is not cleared and the delayed destruction timer has not expired; to destroy the connection management unit and disconnect the audio / video connection if the delayed destruction timer expires and the destruction mark is not cleared; and to clear the destruction mark and maintain the audio / video connection if the connection management unit is found before the delayed destruction timer expires.

[0284] In one exemplary embodiment, the destruction module is further configured to: increment the destruction token of the connection management unit each time a destruction token is set, and record the destruction token when the delayed destruction timer is started; when the delayed destruction timer expires, if the recorded destruction token is consistent with the current destruction token of the connection management unit, execute the operation of destroying the connection management unit and disconnecting the audio and video connection.

[0285] In one exemplary embodiment, the switching module includes an unbinding unit, a page change unit, a tag update unit, and an instruction sending unit, wherein:

[0286] The unbinding unit is used to respond to a switching request and, while maintaining the audio and video connection, release the first binding relationship between the rendered view of the first page and the first audio and video data output by the audio and video connection.

[0287] The page change unit is used to change the currently displayed page of the audio / video connection from the first page to the second page;

[0288] The tag update unit is used to update the mode tag of the audio-video connection from the first mode to the second mode;

[0289] The instruction sending unit is used to send a data request instruction corresponding to the second mode to the target camera device via an audio-visual connection.

[0290] In one exemplary embodiment, the switching module is further configured to: in response to a switching request, close a functional sub-channel established on the audio / video connection; wherein the functional sub-channel includes at least one of a monitoring sub-channel and an intercom sub-channel.

[0291] In one exemplary embodiment, the switching module is further configured to: in response to a switching request, obtain the connection status of the audio / video connection; if the connection status indicates that the audio / video connection has been disconnected, re-establish a connection with the target camera device, and obtain second audio / video data through the re-established connection.

[0292] In one exemplary embodiment, the second display module includes a status notification unit, a second binding unit, and a second display unit, wherein:

[0293] The status notification unit is used to send the current connection status of the audio and video connection to the second page;

[0294] The second binding unit is used to establish a second binding relationship between the rendered view of the second page and the second audio and video data output by the audio and video connection when the current connection status indicates that the audio and video connection is ready.

[0295] The second display unit is used to display the second audio and video data based on the second binding relationship.

[0296] In one exemplary embodiment, the audio / video data mode switching system further includes a preloading module, wherein:

[0297] The preloading module is used to acquire playback event data of the target camera device and write it to the local cache during the process of displaying the first audio and video data on the first page; in response to the switching request, it reads the playback event data from the local cache and displays it on the second page, and requests updated playback event data from the target camera device; if the updated data is obtained, it updates the playback event data displayed on the second page and the local cache.

[0298] In one exemplary embodiment, the target camera device is a first camera device among a plurality of camera devices, and the plurality of camera devices further includes a second camera device; the audio and video data mode switching system further includes a management module, wherein:

[0299] The management module is used to store the connection management units corresponding to multiple camera devices into the connection management unit pool according to the device identifier of each camera device; wherein, each connection management unit maintains the audio and video connection with the corresponding camera device; when the audio and video connection corresponding to the first camera device switches from the first mode to the second mode, the audio and video connection corresponding to the second camera device is maintained.

[0300] The modules in the aforementioned audio and video data mode switching system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module. It should be noted that the above module division is only an example; in specific implementations, two or more modules can be merged into one module, or any module can be split into multiple modules, as long as the implemented functions are equivalent to those of the modules described above.

[0301] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for switching audio and video data modes. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0302] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0303] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0304] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0305] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0306] In one exemplary embodiment, a chip is provided, including a processor and an interface for calling and running a computer program. When the processor calls and runs the computer program through the interface, it implements the steps described in the above method embodiments. This chip can be disposed in a terminal device as a processing component of the terminal device, cooperating with the terminal device's memory, communication interface, and display unit to perform processing such as acquiring audio / video connections, determining the current display page, updating mode markers, sending data request commands, and displaying audio / video data.

[0307] It should be noted that the chip can be a general-purpose processor chip, an application-specific integrated circuit chip, or a system-on-a-chip that includes a processor core and a memory unit; all of the above-mentioned types of chips can be used to implement the audio and video data mode switching method provided in the embodiments of this application, and the embodiments of this application do not limit it.

[0308] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0309] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, memory card (memory chip), optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0310] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0311] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for switching audio and video data modes, characterized in that, A terminal device used for communicating with a target camera device, the method comprising: In response to an access request to the target camera device, an audio-visual connection is obtained; wherein, the audio-visual connection is a connection between the terminal device and the target camera device for transmitting audio-visual data, and the access request is used to request playback of the first audio-visual data in the first mode; Based on the audio and video connection, the first audio and video data is displayed on the first page corresponding to the first mode; In response to a switching request, a second mode corresponding to the switching request is determined, and while maintaining the audio-visual connection, a data request instruction corresponding to the second mode is sent to the target camera device through the audio-visual connection; wherein the first mode and the second mode are different audio-visual data playback modes. The second audio and video data returned by the target camera device based on the data request instruction is obtained through the audio and video connection, and the second audio and video data is displayed on the second page corresponding to the second mode.

2. The method according to claim 1, characterized in that, The step of obtaining the audio / video connection in response to an access request to the target camera device includes: In response to the access request, obtain the device identifier of the target camera device; Check if a connection management unit corresponding to the device identifier exists; wherein, the connection management unit is used to maintain the audio and video connection corresponding to the device identifier; In the absence of the connection management unit, a connection management unit corresponding to the device identifier is created, and the audio and video connection is established with the target camera device through the connection management unit; In the presence of the connection management unit, the connection currently maintained by the connection management unit is determined as the audio / video connection.

3. The method according to claim 1, characterized in that, The step of displaying the first audio and video data on the first page corresponding to the first mode based on the audio and video connection includes: The first page is determined as the current display page of the audio / video connection; Send the current connection status of the audio / video connection to the currently displayed page; When the current connection status indicates that the audio and video connection is ready, a first binding relationship is established between the rendered view of the first page and the first audio and video data output by the audio and video connection; The first audio and video data is displayed based on the first binding relationship.

4. The method according to claim 3, characterized in that, After the step of displaying the first audio and video data on a first page corresponding to the first mode based on the audio and video connection, the method further includes: In response to the exit operation of the currently displayed page, the correspondence between the currently displayed page and the audio / video connection is terminated; Set a destruction marker for the connection management unit and start a delayed destruction timer; The audio and video connection is maintained if the marker to be destroyed is not cleared and the delayed destruction timer has not been reached. If the delayed destruction timer expires and the marker to be destroyed has not been cleared, the connection management unit is destroyed and the audio / video connection is disconnected. If the connection management unit is found before the delayed destruction timer expires, the destruction marker is cleared and the audio / video connection is maintained.

5. The method according to claim 1, characterized in that, The step of responding to a switching request, determining a second mode corresponding to the switching request, and sending a data request instruction corresponding to the second mode to the target camera device through the audio-video connection while maintaining the audio-video connection includes: In response to the switching request, while maintaining the audio and video connection, the first binding relationship between the rendered view of the first page and the first audio and video data output by the audio and video connection is released; Change the current display page of the audio / video connection from the first page to the second page; Update the mode marker of the audio / video connection from the first mode to the second mode; The audio-visual connection is used to send a data request command corresponding to the second mode to the target camera device.

6. The method according to claim 1, characterized in that, The step of displaying the second audio and video data on the second page corresponding to the second mode includes: Send the current connection status of the audio / video connection to the second page; When the current connection status indicates that the audio and video connection is ready, a second binding relationship is established between the rendered view of the second page and the second audio and video data output by the audio and video connection; The second audio and video data is displayed based on the second binding relationship.

7. The method according to any one of claims 1 to 6, characterized in that, The target camera device is a first camera device among a plurality of camera devices, and the plurality of camera devices further includes a second camera device. The method further includes: Each of the multiple camera devices has its corresponding connection management unit stored in a connection management unit pool according to the device identifier of each camera device; wherein each connection management unit maintains an audio and video connection with its corresponding camera device. When the audio / video connection corresponding to the first camera device is switched from the first mode to the second mode, the audio / video connection corresponding to the second camera device is maintained.

8. A mode switching system for audio and video data, characterized in that, A terminal device used for communicating with a target camera device, the system comprising: A connection module is used to obtain an audio-visual connection in response to an access request to the target camera device; wherein the audio-visual connection is a connection between the terminal device and the target camera device for transmitting audio-visual data, and the access request is used to request playback of the first audio-visual data in the first mode; A first display module is configured to display the first audio and video data on a first page corresponding to the first mode, based on the audio and video connection. A switching module is used to respond to a switching request, determine a second mode corresponding to the switching request, and, while maintaining the audio-visual connection, send a data request instruction corresponding to the second mode to the target camera device through the audio-visual connection; wherein, the first mode and the second mode are different audio-visual data playback modes; The second display module is used to obtain the second audio and video data returned by the target camera device based on the data request instruction through the audio and video connection, and to display the second audio and video data on the second page corresponding to the second mode.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the audio and video data mode switching method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images