A screen projection device candidate management method and apparatus, an electronic device, and a computer program product

Abstract

The application discloses a screen projection device candidate management method and device, electronic equipment and computer program product. The method analyzes a sound wave candidate frame to obtain a candidate screen projection code, a candidate network address and a frame identifier, generates a candidate record, and establishes a screen projection code index and a network address index; in response to a device list refresh, the candidate record is queried according to a current discovery address and a service screen projection code, and combined with an invalid time, a network fingerprint, a local network segment, an identity matching condition and an identity conflict, the association is established only when the candidate is unique and there is no conflict. When a connection request occurs, the latest device discovery record is obtained and rechecked, and after the recheck, the connection parameter set is generated and the screen projection connection is established, and if the recheck fails, the association is removed and the connection is prohibited, thereby reducing the risk of false association and false connection in the list refresh, address change and multi-device scene.

Description

Technical Field

[0001] This application relates to the field of screen projection device discovery and connection technology, and in particular to a screen projection device candidate management method, device, electronic device and computer program product. Background Technology

[0002] In screen-sharing initiating terminals such as mobile devices and personal computers, the screen-sharing receiving devices in the current network environment can typically be identified through the local area network service discovery mechanism. Meanwhile, to improve the recognition efficiency of near-field devices, the screen-sharing receiving device can also broadcast a screen-sharing code, network address, or other candidate information via an acoustic channel. The screen-sharing initiating terminal then collects and analyzes the corresponding acoustic signals through a microphone.

[0003] In related technologies, candidate information obtained from acoustic signal parsing may be directly bound to the device list object generated by network service discovery, or directly used as screen mirroring connection parameters. Since the network service discovery module may periodically refresh the device list and replace the original device object with a new one, the acoustic candidate information directly bound to the original device object is easily lost when the device list is refreshed.

[0004] If a device item is added directly to the device list based on the candidate acoustic wave information, in cases of duplicate acoustic wave frames, invalid acoustic wave frames, changes in the network environment, or incorrect parsing of acoustic wave information, a device item that does not correspond to the actual network service discovery device may be generated.

[0005] Furthermore, when multiple screen-casting receiving devices exist within the same local area network, different devices may have the same or similar device names, and the device network addresses may change due to dynamic host configuration, device restarts, or differences in network service discovery timing. Therefore, establishing a correspondence between acoustic candidate information and device discovery results based solely on device name, network address, screen-casting code, or other single fields can easily lead to erroneous candidate associations.

[0006] When a user initiates a screen mirroring connection, the device list interface may still hold old device discovery data due to view reuse, list refresh, or data updates. If a connection is established directly based on old device discovery data or acoustic candidate information that has not been re-verified, it may connect to the wrong device or use an expired network address and connection parameters.

[0007] Therefore, a candidate management scheme is needed that can independently store candidate records for acoustic wave discovery, refresh and restore them across device lists, perform combined identity verification, conflict detection, and re-verification before connection, while acoustic wave discovery and network service discovery can work in parallel. Summary of the Invention

[0008] This application provides a method, apparatus, electronic device, and computer program product for managing screen projection device candidates. It addresses the following technical issues: when directly binding acoustic candidate information to a device list object, refreshing the device list may result in the loss of candidate information; directly adding device discovery records based on acoustic candidate information may generate false device entries; in multi-device scenarios, associating based on a single field can easily lead to incorrect candidate matching; using old device discovery data or expired candidate information when a connection request occurs may lead to connecting to the wrong device; and when acoustic candidate frames and network service discovery data are generated at different times, the connection address and connection parameters may change.

[0009] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for managing candidate screen-casting devices is provided. The method is executed by the screen-casting initiating terminal and includes the following steps.

[0010] Acquire candidate sound wave frames, parse the candidate sound wave frames, and obtain candidate projection codes, candidate network addresses, and frame identifiers.

[0011] Generate candidate records corresponding to the acoustic candidate frames. Each candidate record includes a candidate record identifier, a candidate projection code, a candidate network address, an expiration time, and a network fingerprint. Establish a projection code index with the candidate projection code as the index key, and a network address index with the candidate network address as the index key; both the projection code index and the network address index point to the candidate records.

[0012] In some implementations, a frame identifier index is also established, with the frame identifier as the index key, and the frame identifier index points to the same candidate record. The frame identifier index is used for deduplication of acoustic candidate frames and for recovering candidate records within the same acoustic acquisition period, but is not used to separately determine the identity correspondence between candidate records and device discovery records.

[0013] In response to a device list refresh event triggered by the network service discovery module, at least one device discovery record is retrieved. The device discovery record includes a device discovery record identifier, the current discovery address, and the service casting code.

[0014] Use at least one of the currently discovered addresses and service casting codes to query the casting code index and network address index to obtain a set of candidate records.

[0015] The current local network segment and current network fingerprint are determined based on the current active network interface of the screen projection initiating terminal, and candidate review processing is performed on the candidate records in the candidate record set.

[0016] The candidate verification process includes verifying the expiration time and network fingerprint of the candidate record, and determining whether the candidate record meets the identity matching conditions based on the candidate network address, current discovery address, candidate projection code, service projection code, and optional enhanced identity fields, while also detecting whether there is an identity conflict.

[0017] In conjunction with the first aspect mentioned above, in one possible implementation, the candidate network address is the same as the currently discovered address, both the candidate network address and the currently discovered address belong to the current local network segment, and they are the same when both the candidate screen projection code and the service screen projection code exist, thereby determining that the candidate record meets the identity matching conditions.

[0018] In conjunction with the first aspect mentioned above, one possible implementation involves the candidate projection code being identical to the service projection code, and both the candidate network address and the currently discovered address belonging to the current local network segment. This determines that the candidate record meets the identity matching criteria. This method can continue to identify the same projection receiving device even when the device's network address changes but the projection code remains unchanged.

[0019] An identity conflict is determined when both the candidate casting code and the service casting code exist and are different. An identity conflict is also determined when a candidate network address has already established a unique correspondence with a device discovery record different from the current device discovery record.

[0020] Only when there is one candidate record in the candidate record set that meets the identity matching condition and there is no identity conflict, is an association established between the device discovery record identifier and the candidate record identifier.

[0021] After establishing the association, the candidate record identifier can be written into the data structure corresponding to the device discovery record, and the display status of the acoustic direct connection identifier for that device discovery record can be set. All device discovery records in the device list are generated by the network service discovery module; no new device discovery records are added to the device list based on acoustic candidate frames.

[0022] The association is not established if the candidate network address and the currently discovered address belong to the current local network segment, the device name is the same, the candidate record is found only through the frame identifier index, or multiple device discovery records meet the auxiliary matching conditions with the same candidate record.

[0023] In response to a connection request for a device discovery record, the latest device discovery record is obtained according to the device discovery record identifier, and candidate records are restored based on the association, projection code index, or network address index.

[0024] After obtaining the latest device discovery record, the candidate verification process is re-executed based on this record. The re-executed candidate verification process includes expiration time verification, network fingerprint verification, identity matching judgment, and identity conflict detection.

[0025] If the result of re-performing the candidate review process is that only one candidate record meets the identity matching condition and there is no identity conflict, a connection parameter set for establishing the screen projection connection is generated.

[0026] If the latest device discovery record contains the current discovery address, the current discovery address in the latest device discovery record will be used as the connection address.

[0027] If the latest device discovery record does not contain the currently discovered address, and the candidate network address is still within its expiration period, belongs to the current local network segment, and does not have an identity conflict, the candidate network address will be used as the connection address.

[0028] The connection parameter set includes a connection address, a candidate record identifier, a network fingerprint, and a verification time. In some embodiments, the connection parameter set also includes an address source marker, which indicates that the connection address comes from the latest device discovery record or from an acoustic candidate frame.

[0029] In some implementations, generating the connection parameter set includes generating an acoustic connection snapshot. The acoustic connection snapshot includes the connection address, address source marker, candidate record identifier, candidate projection code digest, network fingerprint, and verification time. The acoustic connection snapshot is sent as a set of connection parameters to the streaming media connection module.

[0030] The streaming media connection module establishes a screen mirroring connection based on the connection parameters in the acoustic connection snapshot. If a change in the current network fingerprint is detected or the candidate verification process fails, the acoustic connection snapshot is set to a disabled state.

[0031] If the candidate review process fails to pass, the association between the device discovery record identifier and the candidate record identifier is severed, the corresponding acoustic direct connection identifier is cleared, and screen projection connection based on the candidate record is prohibited.

[0032] In conjunction with the first aspect above, in one possible implementation, when another acoustic candidate frame with the same frame identifier is received, the acoustic candidate frame is determined to be a duplicate acoustic candidate frame according to the frame identifier index, and the most recent hit time of the candidate record is updated, without generating duplicate candidate records.

[0033] Frame identifiers can be used to identify acoustic frames, acoustic transmission periods, or acoustic acquisition periods. Since frame identifiers may change with the transmission period, the frame identifier index is not used as a permanent identifier for the projection receiving device, nor is it used alone to establish a relationship between candidate records and device discovery records.

[0034] In conjunction with the first aspect mentioned above, in one possible implementation, the candidate record also includes the currently bound address. When generating a candidate record, the currently bound address is set as the candidate network address.

[0035] When the candidate projection code is the same as the service projection code, the candidate network address and the current discovery address both belong to the current local network segment, there is no identity conflict, and the current discovery address is different from the current binding address, delete the network address index entry that uses the current binding address before the update as the index key and points to the candidate record, create a network address index entry that uses the current discovery address as the index key and points to the candidate record, and update the current binding address in the candidate record to the current discovery address.

[0036] By updating the current bound address and network address index, the candidate record identifier can remain unchanged and the same candidate record can continue to be restored when the address of the screen-casting receiving device changes due to dynamic host configuration address renewal or network service discovery timing differences.

[0037] In conjunction with the first aspect above, in one possible implementation, when the current network fingerprint does not match the network fingerprint in the candidate record, and it is impossible to determine that the candidate record uniquely corresponds to the latest device discovery record based on at least one of the candidate projection code, device persistent identifier, or service instance name, the candidate record is set to an invalid state.

[0038] When the candidate projection code, candidate network address, and frame identifier all change, the original candidate record can also be set to an invalid state.

[0039] Once a candidate record is set to an invalid state, the projection code index entry and network address index entry pointing to that candidate record are deleted. In implementations with a frame identifier index, the corresponding frame identifier index entry can also be deleted. Deleted index entries must not be used to restore candidate records.

[0040] The system can wait for new acoustic candidate frames to arrive and regenerate candidate records and corresponding indexes based on the new acoustic candidate frames.

[0041] By implementing the above failure handling, we can avoid using old candidate records when the screen casting initiating terminal switches networks, the screen casting receiving device restarts, or the overall candidate identity information changes.

[0042] In conjunction with the first aspect above, in one possible implementation, the candidate record and the device discovery record each include at least one enhanced identity field selected from the service instance name, the device persistent identifier, or the certificate public key fingerprint.

[0043] If the candidate record is within its expiration time, the network fingerprint in the candidate record matches the current network fingerprint, and there is no identity conflict, the enhanced identity field in the candidate record is compared with the corresponding enhanced identity field in the device discovery record.

[0044] When the corresponding enhanced identity fields are the same, the comparison result of the enhanced identity fields can be used to determine whether the candidate record meets the identity matching condition.

[0045] The enhanced identity field is used as supplementary evidence for identity matching, and does not replace expiration time verification, network fingerprint verification, or identity conflict detection.

[0046] In a second aspect, a screen projection device candidate management device includes an acoustic frame processing module, a candidate record management module, a device discovery module, a candidate association module, a connection verification module, and a connection parameter module.

[0047] The acoustic frame processing module is configured to acquire acoustic candidate frames and parse them to obtain candidate projection codes, candidate network addresses, and frame identifiers.

[0048] The candidate record management module is configured to generate candidate records. Each candidate record includes a candidate record identifier, a candidate projection code, a candidate network address, an expiration time, and a network fingerprint. The candidate record management module is also configured to establish projection code indexes and network address indexes, both pointing to the candidate records.

[0049] The device discovery module is configured to obtain at least one device discovery record through network service discovery. The device discovery record includes a device discovery record identifier, the current discovery address, and the service projection code.

[0050] The candidate association module is configured to query the projection code index and network address index using at least one of the current discovery address and service projection code to obtain a set of candidate records; determine the current local network segment and current network fingerprint based on the currently active network interface; perform expiration time verification, network fingerprint verification, identity matching, and identity conflict detection on the candidate records in the candidate record set; and establish an association between the device discovery record identifier and the candidate record identifier when only one candidate record passes the corresponding verification and identity matching and no identity conflict is detected.

[0051] The connection verification module is configured to respond to connection requests for device discovery records by obtaining the latest device discovery record based on the device discovery record identifier, restoring the corresponding candidate record, and re-performing expiration time verification, network fingerprint verification, identity matching, and identity conflict detection based on the latest device discovery record.

[0052] The connection parameter module is configured to determine the connection address, generate a connection parameter set including the connection address, candidate record identifier, network fingerprint and verification time, and send the connection parameter set to the streaming media connection module when the connection verification module determines that only one candidate record has passed the verification and no identity conflict has been detected.

[0053] In conjunction with the second aspect above, in one possible implementation, the candidate record management module is further configured to establish a frame identifier index with the frame identifier as the index key and pointing to the candidate record, and to deduplicate the duplicate acoustic wave candidate frames according to the frame identifier index.

[0054] In conjunction with the second aspect above, in one possible implementation, the candidate record management module is further configured to, when the candidate projection code is the same as the service projection code, the candidate network address and the current discovery address both belong to the current local network segment and there is no identity conflict, delete the network address index entry with the current binding address before the update as the index key, create a network address index entry with the current discovery address as the index key, and update the current binding address in the candidate record.

[0055] In conjunction with the second aspect described above, in one possible implementation, the device further includes a candidate protection module. The candidate protection module is configured to detect at least one of the following abnormal conditions: candidate record expiration time, network fingerprint mismatch, candidate projection code not matching the service projection code, or candidate network address establishing a unique correspondence with another device discovery record.

[0056] When abnormal conditions are detected, the candidate protection module outputs a prohibition association command to the candidate association module and the connection parameter module to remove the association between the device discovery record identifier and the candidate record identifier, and to prevent the generation of the connection parameter set used to establish the screen projection connection.

[0057] Thirdly, this application provides a screen mirroring initiating electronic device, including an audio input interface, a network communication interface, a memory, and a processor.

[0058] The audio input interface is used to receive sound wave signals collected by the microphone.

[0059] The network communication interface is used to receive network service discovery data and send screen mirroring connection data.

[0060] The memory is used to store computer programs, as well as candidate records, projection code indexes, network address indexes, and optional frame identifier indexes.

[0061] The processor is connected to the audio input interface, the network communication interface, and the memory. When the processor executes the computer program, it implements the screen projection device candidate management method provided in this application.

[0062] Fourthly, this application provides a computer program product, including a computer program. When the computer program runs on a processor, it enables the processor to implement the screen projection device candidate management method provided in this application.

[0063] The computer program may be stored or carried on a non-volatile computer-readable storage medium. The computer-readable storage medium may include a read-only memory, random access memory, solid-state memory, flash memory, magnetic storage medium, optical storage medium, or other media capable of storing a computer program.

[0064] Beneficial effects: 1. By establishing a projection code index and a network address index, and optionally a frame identifier index, multiple indexes can be made to point to the same candidate record. After the network service discovery device list is refreshed, candidate records can be restored based on the current discovery address or service projection code in the newly generated device discovery record, avoiding the dependency of candidate information on a specific device list object.

[0065] 2. By using the current local network segment, expiration time, and network fingerprint as candidate verification conditions, and combining them with candidate network addresses, current discovery addresses, candidate projection codes, service projection codes, and enhanced identity fields for identity matching, the risk of mismatch caused by associating based on a single field can be reduced.

[0066] 3. By performing identity conflict handling when candidate projection codes conflict, candidate network address ownership conflicts, or multiple candidate records simultaneously meet matching conditions, the association relationship can be established only when a unique candidate record meets the identity matching conditions and there is no identity conflict. This improves the accuracy of association in multi-device environments. By only writing the candidate record identifier into the device discovery record generated by the network service discovery module, instead of adding new device discovery records based on acoustic candidate frames, false device entries caused by duplicate, expired, or parsed acoustic frames can be avoided. By obtaining the latest device discovery record and re-performing candidate verification processing when a connection request occurs, the use of old data before the device list was refreshed or expired candidate records to establish projection connections can be avoided.

[0067] 4. By prioritizing the use of currently discovered addresses from the latest device discovery records, and only using candidate network addresses when they are missing and meet the conditions of validity period, same network segment, and no conflict, the timeliness and reliability of connection addresses can be improved. By generating an acoustic connection snapshot containing the connection address, address source tag, candidate record identifier, screen casting code digest, network fingerprint, and verification time, and stopping the use of this snapshot when the network environment changes or verification fails, the consistency of connection parameters during screen casting connection establishment can be maintained. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the overall architecture of the screen projection device candidate management device provided in the embodiments of this application.

[0069] Figure 2 This is a schematic diagram of the candidate record index and its association relationship provided in the embodiments of this application.

[0070] Figure 3 This is a schematic diagram of the candidate association process when refreshing the device list, as provided in an embodiment of this application.

[0071] Figure 4 This is a schematic diagram of the candidate review and connection control flow for connection requests provided in an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0073] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0074] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0075] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0076] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0077] The screen projection device candidate management method provided in this application embodiment can be executed by a screen projection device candidate management device. The screen projection device candidate management device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. The screen projection device candidate management device can be located in a screen projection sending end, a screen projection receiving end, or other electronic devices communicatively connected to the screen projection sending end. The electronic devices can be mobile phones, tablets, laptops, desktop computers, interactive whiteboards, conference terminals, screen projectors, or other intelligent devices with data processing and communication functions.

[0078] For ease of understanding, this application uses a screen projector as an example to describe the application carrier of a candidate management device for screen projecting equipment. The screen projector can be an electronic device used to establish a communication connection between the screen projector and the display device, and to transmit image data, audio data, video data, control data, or other data to be projected from the screen projector to the display device. The screen projector can be set up independently or integrated into a monitor, television, interactive whiteboard, conference terminal, projection device, or other display device.

[0079] A screen mirroring device can function as a screen mirroring sender, transmitting locally generated or acquired data from other devices to a screen mirroring receiver. It can also function as a screen mirroring receiver, receiving screen mirroring data from mobile phones, tablets, laptops, or other screen mirroring senders, and outputting that data to a display device connected to the screen mirroring device. Furthermore, a screen mirroring device can simultaneously perform both screen mirroring sending and receiving functions.

[0080] Generally, a screen mirroring device may include at least one of a processor, memory, communication module, and display interface. The communication module may include a wireless communication module and / or a wired communication module. The wireless communication module may support wireless LAN, Bluetooth, ultra-wideband, near-field communication, mobile communication, acoustic communication, or other wireless communication methods; the wired communication module may support Ethernet, Universal Serial Bus, or other wired communication methods. The display interface may include a high-definition multimedia interface, a DisplayPort interface, a Universal Serial Bus interface, a video graphics array interface, or other interfaces capable of transmitting image and / or audio data.

[0081] The screen mirroring device can access a local area network (LAN) via Wi-Fi and search for available screen mirroring receivers within the LAN using a network service discovery mechanism. The device can receive device discovery messages broadcast by one or more screen mirroring receivers within the LAN and, based on these messages, obtain at least one of the following: device name, device identifier, network address, service type, port number, connection status, and device capabilities, thereby generating or updating the list of available screen mirroring devices.

[0082] The screen mirroring device can also be equipped with a microphone, a sound wave receiving module, or other sound acquisition modules to collect sound wave signals sent by the screen mirroring receiver. The screen mirroring device can demodulate and analyze the collected sound wave signals to obtain sound wave candidate information. The sound wave candidate information may include at least one of the following: screen mirroring code, network address, device identifier, device name, timestamp, verification information, and sound wave signal strength. The screen mirroring device can associate the sound wave candidate information with the corresponding device in the list of eligible screen mirroring devices based on the matching relationship between the sound wave candidate information and the device information obtained through a network service discovery mechanism.

[0083] The screen mirroring device can establish an association between candidate acoustic wave information and screen-sharing devices based on a device identifier, network address, screen mirroring code, device name, or a combination of at least two of the above information. This association can be stored independently of the list row objects in the screen-sharing device list, so that even when the screen-sharing device list is refreshed, the list row objects are rebuilt, or the device display order changes, the candidate acoustic wave information can still be re-associated to the corresponding screen-sharing device based on the combined identifier.

[0084] Typically, the screen mirroring device can install at least one operating system, including but not limited to Android, Linux, Windows, HarmonyOS, or an embedded real-time operating system. The screen mirroring device can run at least one screen mirroring application based on the operating system. The screen mirroring application can be an application that comes pre-installed with the operating system, or an application downloaded and installed from a server or other electronic device. The screen mirroring device candidate management device can be a standalone application or implemented as a functional module within a screen mirroring application.

[0085] The screen mirroring application can perform at least one of the following operations: device discovery, device list generation, acoustic signal parsing, candidate information caching, device matching, connection request sending, screen mirroring code verification, screen mirroring link establishment, and screen mirroring status update. For example, a user can select a target screen mirroring device from the list of available devices. The screen mirroring device obtains the corresponding network address and / or screen mirroring code based on the selected target device, sends a connection request to the target device, and establishes a screen mirroring communication link after successful connection verification.

[0086] Understandably, users can select a target projection device or initiate a projection operation via touchscreen, keyboard, mouse, remote control, physical buttons, voice commands, or other input methods. After receiving the user's operation, the projection device can determine the target projection device and its corresponding connection parameters based on the user's selected device information, acoustic wave candidate information, and the current network connection status, and then execute the projection connection operation.

[0087] The projection device may include one or more processing cores. The processing core can implement the projection device candidate management method provided in this application embodiment using pure software, or it can be implemented using a combination of software and hardware. For example, it can be implemented using at least one hardware form selected from digital signal processors, field-programmable gate arrays, application-specific integrated circuits, and programmable logic arrays. The projection device may also integrate one or more of a central processing unit, a graphics processor, a network processor, an audio processor, and a modem.

[0088] The memory of the screen projector can be used to store at least one of the following: computer program, list of projectable devices, device discovery information, acoustic candidate information, device association relationships, screen projecting connection parameters, and screen projecting status information. The processor can call and execute the computer program stored in the memory to cause the screen projector to execute the screen projecting device candidate management method provided in the embodiments of this application.

[0089] The computer program can be presented in a form suitable for the operation of the screen mirroring device, such as an application, system service, background process, system plugin, software development kit, firmware program, or web page plugin. In some embodiments, the screen mirroring device candidate management method can also be executed collaboratively by multiple software modules in the screen mirroring device. The multiple software modules may include at least one of a device discovery module, a sound wave analysis module, a candidate information management module, a device matching module, a list update module, and a connection control module.

[0090] Example 1 like Figure 1 As shown, the screen projection device candidate management device can be deployed in the screen projection initiating terminal, including an acoustic frame processing module, a candidate record management module, a device discovery module, a candidate association module, a connection verification module, a connection parameter module, and a candidate protection module.

[0091] The screen mirroring initiating terminal receives the sound wave signal collected by the microphone through the audio input interface. The sound wave frame processing module obtains the sound wave candidate frames from the sound wave signal and parses them to obtain the candidate screen mirroring code, candidate network address, and frame identifier.

[0092] The candidate record management module generates candidate record identifiers for acoustic wave candidate frames and generates corresponding candidate records. Candidate records may include candidate record identifiers, candidate projection codes, candidate network addresses, frame identifiers, first hit time, most recent hit time, expiration time, network fingerprints, and currently bound addresses.

[0093] The candidate record management module establishes a screen-casting code index and a network address index. The screen-casting code index uses the candidate screen-casting code as the index key, and the network address index uses the candidate network address or the currently bound address as the index key. Both indexes point to the same candidate record.

[0094] In the implementation of setting the frame identifier index, the candidate record management module also establishes a frame identifier index with the frame identifier as the index key, and the frame identifier index also points to the candidate record.

[0095] When a candidate acoustic frame with the same frame identifier is received again, the candidate record management module queries the existing candidate record through the frame identifier index and updates the most recent hit time of the candidate record, instead of creating the candidate record again.

[0096] The device discovery module receives network service discovery data through the network communication interface and generates device discovery records based on the network service discovery data. Each device discovery record may include at least one of the following: device discovery record identifier, current discovery address, service projection code, device name, service instance name, device persistent identifier, or certificate public key fingerprint.

[0097] Candidate records and their corresponding indexes are stored independently of device discovery records. Therefore, even if the network service discovery module refreshes the device list and regenerates the device discovery record object, the candidate records will not disappear because the original device row object has been replaced.

[0098] Example 2 like Figure 2 As shown, the projection code index, network address index, and frame identifier index provide different query entry points for candidate records, and all point to the candidate records.

[0099] The screen projection code index is used to query candidate records based on candidate screen projection codes or service screen projection codes. The network address index is used to query candidate records based on candidate network addresses, currently bound addresses, or currently discovered addresses.

[0100] The frame identifier index is used to detect repeating acoustic candidate frames and to recover candidate records within the same acoustic acquisition period. The frame identifier index is not used to individually confirm that a candidate record and a device discovery record belong to the same projection receiving device.

[0101] Candidate records and device discovery records are independent data records. Once the candidate review process confirms that the candidate record and device discovery record uniquely correspond and there are no identity conflicts, an association is established between the device discovery record identifier and the candidate record identifier.

[0102] This association can be stored in a separate association table, or it can be implemented by writing the candidate record identifier into the data structure of the device discovery record.

[0103] The network service discovery module remains the primary generator of device discovery records in the device list. The acoustic frame processing module and the candidate record management module do not directly add device discovery records to the device list based on acoustic candidate frames.

[0104] When a connection request occurs, the connection verification module can prioritize restoring candidate records based on the association relationship. If the association relationship is missing or the device discovery record is regenerated, it can also restore candidate records using the current discovery address or service projection code in the latest device discovery record, through network address index or projection code index.

[0105] After the connection verification is successful, the connection parameter module generates an acoustic connection snapshot. The acoustic connection snapshot is associated with the candidate record and the device discovery record, but it is sent to the streaming media connection module independently as a whole as the connection parameters.

[0106] Example 3 like Figure 3 As shown, when the network service discovery module triggers a device list refresh event, the candidate association module obtains at least one device discovery record from the current device list.

[0107] For each device discovery record, the candidate association module reads the current discovery address and service projection code, and uses at least one of the current discovery address and service projection code to query the network address index and projection code index to obtain a set of candidate records.

[0108] The query index is only used to obtain candidate records that may correspond to the device discovery record, and does not mean that the candidate records matched by the query have been identified and matched with the device discovery record.

[0109] The candidate association module determines the current local network segment and the current network fingerprint based on the current active network interface of the screen projection initiating terminal, and performs candidate review processing on the candidate record set.

[0110] First, the candidate association module determines whether the candidate record is within its expiration period and whether the network fingerprint in the candidate record matches the current network fingerprint. Candidate records that do not meet the expiration or network fingerprint conditions cannot be used to establish association relationships.

[0111] Subsequently, the candidate association module determines whether the candidate record meets the identity matching conditions.

[0112] In the first scenario, if the candidate network address is the same as the currently discovered address, both the candidate network address and the currently discovered address belong to the current local network segment, and both the candidate screen projection code and the service screen projection code are the same, then the candidate record meets the identity matching conditions.

[0113] In the second case, if the candidate projection code is the same as the service projection code, and both the candidate network address and the currently discovered address belong to the current local network segment, then the candidate record meets the identity matching condition.

[0114] When both candidate records and device discovery records include enhanced identity fields, the corresponding enhanced identity fields can also be compared. When the enhanced identity fields are the same, and the expiration time, network fingerprint, and identity conflict detection all pass, the comparison result of the enhanced identity fields can be used as evidence of identity matching.

[0115] The candidate association module also performs identity conflict detection. An identity conflict is determined when both the candidate casting code and the service casting code exist and are different. An identity conflict is also determined when a candidate network address has already established a unique correspondence with another device discovery record.

[0116] The same network segment condition is only a prerequisite for security and cannot be used alone to prove that a candidate record corresponds to a device discovery record. Similarly, identical device names, matching frame identifiers only, or multiple device discovery records simultaneously meeting auxiliary matching conditions cannot be used alone to establish a relationship.

[0117] When only one candidate record meets the identity matching condition and there is no identity conflict, the candidate association module establishes the association relationship between the device discovery record identifier and the candidate record identifier, writes the candidate record identifier into the data structure corresponding to the device discovery record, and sets the display status of the acoustic direct connection identifier of the device discovery record.

[0118] When no candidate record meets the identity matching condition, multiple candidate records meet the identity matching condition simultaneously, or there is an identity conflict, no association is established and no direct connection identifier is displayed.

[0119] If the candidate projection code is the same as the service projection code, the candidate network address and the currently discovered address both belong to the current local network segment, there is no identity conflict, and the currently discovered address is different from the currently bound address in the candidate record, then the candidate record management module will perform an address index update.

[0120] Specifically, the candidate record management module deletes the network address index entry that uses the original currently bound address as the index key and points to the candidate record, creates a network address index entry that uses the current discovery address as the index key and points to the candidate record, and updates the current bound address in the candidate record to the current discovery address.

[0121] Therefore, even if the network address of the receiving device changes but the casting code or enhanced identity field remains stable, the correspondence between candidate records and device discovery records can continue to be maintained.

[0122] Example 4 like Figure 4 As shown, when a user initiates a connection request for a device discovery record, the connection verification module obtains the latest device discovery record based on the device discovery record identifier.

[0123] The device discovery record identifier can be a stable record identifier generated by the network service discovery module, or it can be an identifier generated by the screen mirroring application based on service instance information, persistent device identifiers, or other stable information.

[0124] The connection verification module does not directly use the old device row object held by the interface component, but instead obtains the latest device discovery record from the current device discovery data based on the device discovery record identifier.

[0125] The connection verification module first reads the candidate record identifier written in the latest device discovery record, and then restores the candidate record based on the candidate record identifier.

[0126] When the latest device discovery record does not contain a candidate record identifier, the connection verification module queries the screen projection code index and network address index based on the service projection code or the current discovery address in the latest device discovery record to restore the candidate record set.

[0127] After restoring the candidate records, the connection review module re-executes the candidate review process, including expiration time verification, network fingerprint verification, identity matching judgment, and identity conflict detection.

[0128] The connection parameter module generates a connection parameter set only when the result of re-executing the candidate review process is that only one candidate record meets the identity matching condition and there is no identity conflict.

[0129] When the latest device discovery record contains the current discovery address, the connection parameter module prioritizes determining the current discovery address as the connection address and sets the address source flag to a value indicating the source of the network service discovery.

[0130] When the latest device discovery record does not contain the currently discovered address, the connection parameter module further determines whether the candidate network address is within the expiration time, whether it belongs to the current local network segment, and whether there is an identity conflict.

[0131] When all the above conditions are met, the candidate network address is determined as the connection address, and the address source flag is set to the value indicating the source of the acoustic candidate frame.

[0132] If the latest device discovery record does not contain the currently discovered address, and the candidate network address does not meet the conditions of validity period, same network segment, or no conflict, a connection parameter set for establishing a screen mirroring connection will not be generated.

[0133] The connection parameter module generates an acoustic connection snapshot based on the final determined connection address. The acoustic connection snapshot includes the connection address, address source marker, candidate record identifier, candidate projection code digest, network fingerprint, and verification time.

[0134] The digest of the candidate projection code can be generated using a hash algorithm or other digest algorithm to avoid transmitting the complete candidate projection code between modules. The verification time indicates the time it takes for the connection parameters to pass through the candidate verification process.

[0135] The connection parameter module sends the acoustic connection snapshot as a set of connection parameters to the streaming media connection module. The streaming media connection module establishes a projection connection based on the acoustic connection snapshot and uses the parameters from the same acoustic connection snapshot during the connection establishment process.

[0136] When the current network fingerprint changes, the candidate record expires, the candidate review process fails, or an identity conflict is detected, the acoustic connection snapshot will be set to a stopped state.

[0137] When the candidate review process fails to pass, the connection review module removes the association between the device discovery record identifier and the candidate record identifier, clears the acoustic direct connection identifier of the device discovery record, and prohibits the establishment of a screen projection connection based on the candidate record.

[0138] After switching networks on the device that initiated the screen mirroring, the current network fingerprint may not match the network fingerprint in the candidate records.

[0139] When the network fingerprint does not match, but the candidate projection code, device persistent identifier, service instance name, or certificate public key fingerprint can uniquely prove that the candidate record corresponds to the latest device discovery record, the candidate record can be reconfirmed according to the preset policy.

[0140] When the network fingerprint does not match and a unique correspondence cannot be determined based on at least one of the candidate projection code, device persistent identifier, or service instance name, the candidate record is set to an invalid state.

[0141] When the candidate projection code, candidate network address, and frame identifier all change, it can be assumed that the original candidate record can no longer represent the current projection receiving device, and the original candidate record is set to an invalid state.

[0142] After a candidate record becomes invalid, the candidate record management module deletes the projection code index entry and network address index entry pointing to that candidate record. If a frame identifier index is set, the corresponding frame identifier index entry can also be deleted.

[0143] After an index entry is deleted, candidate records are not restored based on the deleted index entry. When a new acoustic candidate frame is received subsequently, candidate records and corresponding indexes are regenerated.

[0144] The candidate protection module can simultaneously send a prohibition command to both the candidate association module and the connection parameter module, causing the candidate association module to stop establishing association relationships and the connection parameter module to stop generating connection parameter sets.

[0145] Example 5 This application also provides a computer-readable storage medium storing a computer program thereon, the instructions of which are adapted to be loaded by a processor and executed by the method steps of Embodiment 1 shown above. The specific execution process can be found in the detailed description of the embodiments, and will not be repeated here. The device containing the storage medium can be an electronic device such as a personal computer, laptop computer, smartphone, or tablet computer.

[0146] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function selected in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function selected in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for managing candidate projection devices, characterized in that, The method, executed by the terminal initiating the screen mirroring, includes: Acquire candidate sound waves, and parse the candidate sound waves to obtain candidate projection codes, candidate network addresses, and frame identifiers; Generate a candidate record corresponding to the acoustic candidate frame. The candidate record includes a candidate record identifier, the candidate projection code, the candidate network address, the expiration time, and the network fingerprint. Establish a projection code index with the candidate projection code as the index key and a network address index with the candidate network address as the index key. Both the projection code index and the network address index point to the candidate record. In response to a device list refresh event triggered by the network service discovery module, at least one device discovery record is obtained, the device discovery record including a device discovery record identifier, the current discovery address, and the service projection code; Query the screen casting code index and the network address index using at least one of the current discovery address and the service screen casting code to obtain a candidate record set; The current local network segment and current network fingerprint are determined based on the currently active network interface of the screen projection initiating terminal, and candidate review processing is performed on the candidate records in the candidate record set. The candidate review processing includes: Verify whether the candidate record is within the expiration time and whether the network fingerprint in the candidate record matches the current network fingerprint; If the candidate network address is the same as the currently discovered address, both the candidate network address and the currently discovered address belong to the current local network segment, and both the candidate screen projection code and the service screen projection code are the same when both exist, or if the candidate screen projection code is the same as the service screen projection code and both the candidate network address and the currently discovered address belong to the current local network segment, then the candidate record is determined to meet the identity matching conditions. If both the candidate screen projection code and the service screen projection code exist and are different, or if the candidate network address has been uniquely associated with another device discovery record that is different from the device discovery record, an identity conflict is determined to exist. If only one candidate record satisfies the identity matching condition and there is no identity conflict, establish an association between the device discovery record identifier and the candidate record identifier; In response to a connection request for the device discovery record, the latest device discovery record is obtained according to the device discovery record identifier, and the candidate record is restored based on the association, the projection code index, or the network address index. The candidate review process will be re-executed based on the latest device discovery record; If the result of re-executing the candidate review process is that only one candidate record meets the identity matching condition and there is no identity conflict, the current discovery address in the latest device discovery record is used as the connection address to generate a connection parameter set including the connection address, the candidate record identifier, the network fingerprint and the review time, and the connection parameter set is sent to the streaming media connection module so that the streaming media connection module establishes a screen casting connection. If the candidate review process fails to pass, the association is terminated and the screen projection connection based on the candidate record is prohibited.

2. The screen projection device candidate management method according to claim 1, characterized in that, It also includes establishing a frame identifier index with the frame identifier as the index key, the frame identifier index pointing to the candidate record; When another acoustic candidate frame with the same frame identifier is received, the other acoustic candidate frame is determined to be a duplicate acoustic candidate frame according to the frame identifier index, and the most recent hit time of the candidate record is updated. The frame identifier index is used for deduplication of acoustic candidate frames and recovery of candidate records within the same acoustic acquisition cycle, but is not used to determine the identity correspondence between the candidate record and the device discovery record.

3. The method for managing candidate projection devices according to claim 1, characterized in that, The candidate record also includes the current binding address, which is set as the candidate network address when the candidate record is generated; If the candidate projection code is the same as the service projection code, the candidate network address and the currently discovered address both belong to the current local network segment and there is no identity conflict, and the currently discovered address is different from the currently bound address: Delete the network address index entry that uses the currently bound address as the index key and points to the candidate record; Create a network address index entry that uses the currently discovered address as the index key and points to the candidate record; as well as Update the current binding address in the candidate record to the current discovery address.

4. The screen projection device candidate management method according to claim 1, characterized in that, The association will not be established if the candidate record and the device discovery record meet only one of the following conditions: Both the candidate network address and the currently discovered address belong to the current local network segment; The device name corresponding to the candidate record is the same as the device name corresponding to the device discovery record; The candidate record is retrieved only through the frame identifier index; or At least two devices found records that met the preset auxiliary matching conditions with the same candidate record.

5. The screen projection device candidate management method according to claim 1, characterized in that, If the current network fingerprint does not match the network fingerprint in the candidate record, and it cannot be determined that the candidate record uniquely corresponds to the latest device discovery record based on at least one of the candidate projection code, device persistent identifier, or service instance name, or if the candidate projection code, the candidate network address, and the frame identifier have all changed: Set the candidate record to an invalid state; Delete the screen capture code index entry and network address index entry pointing to the candidate record; and Recovering the candidate record based on a deleted index entry is prohibited.

6. The screen projection device candidate management method according to claim 1, characterized in that, If the latest device discovery record does not contain the currently discovered address, and the candidate network address is still within the expiration time, belongs to the current local network segment, and does not have the identity conflict, the candidate network address shall be used as the connection address. The connection parameter set also includes an address source marker indicating that the connection address comes from the latest device discovery record or the acoustic candidate frame.

7. The method for managing candidate projection devices according to claim 1, characterized in that, Generating the connection parameter set includes generating an acoustic connection snapshot, which includes the connection address, address source marker, candidate record identifier, digest of the candidate projection code, network fingerprint, and verification time. Sending the connection parameter set to the streaming media connection module includes sending the acoustic connection snapshot as a whole set of connection parameters to the streaming media connection module; If a change in the current network fingerprint is detected or the candidate verification process fails, the acoustic connection snapshot will be set to a stopped state.

8. The method for managing candidate projection devices according to claim 1, characterized in that, The candidate record and the device discovery record each include at least one enhanced identity field among the service instance name, device persistent identifier, or certificate public key fingerprint; If the candidate record is within the expiration time, the network fingerprint matches, and there is no identity conflict, the enhanced identity field in the candidate record is compared with the corresponding enhanced identity field in the device discovery record. If both are the same, the comparison result of the enhanced identity field is used to determine whether the candidate record meets the identity matching condition.

9. The screen projection device candidate management method according to claim 1, characterized in that, After establishing the association, the candidate record identifier is written into the data structure corresponding to the device discovery record, and the display status of the acoustic direct connection identifier of the device discovery record is set. The device discovery records in the device list are all generated by the network service discovery module, and no new device discovery records are added to the device list based on the acoustic candidate frames.

10. A candidate management device for screen projection devices, characterized in that, include: The acoustic frame processing module is configured to acquire acoustic candidate frames and parse the acoustic candidate frames to obtain candidate projection codes, candidate network addresses and frame identifiers. The candidate record management module is configured to generate candidate records including candidate record identifier, candidate projection code, candidate network address, expiration time and network fingerprint, and to establish projection code index and network address index that both point to the candidate record; The device discovery module is configured to obtain at least one device discovery record through network service discovery, the device discovery record including a device discovery record identifier, the current discovery address and the service projection code; The candidate association module is configured as follows: Query the screen casting code index and the network address index using at least one of the current discovery address and the service screen casting code to obtain a candidate record set; Determine the current local network segment and the current network fingerprint based on the currently active network interface; Expiration time verification, network fingerprint verification, identity matching, and identity conflict detection are performed on the candidate records in the candidate record set; If only one candidate record passes the expiration time verification, the network fingerprint verification, and the identity matching, and no identity conflict is detected, an association is established between the device discovery record identifier and the corresponding candidate record identifier. The connection verification module is configured to, in response to a connection request for the device discovery record, obtain the latest device discovery record based on the device discovery record identifier, restore the corresponding candidate record, and re-perform the expiration time verification, network fingerprint verification, identity matching, and identity conflict detection based on the latest device discovery record; as well as The connection parameter module is configured to, when the connection verification module determines that only one candidate record passes the verification and no identity conflict is detected, use the current discovery address in the latest device discovery record as the connection address, generate a connection parameter set including the connection address, the candidate record identifier, the network fingerprint and the verification time, and send the connection parameter set to the streaming media connection module.

11. The candidate management device for projection devices according to claim 10, characterized in that, The candidate record management module is also configured as follows: Establish a frame identifier index with the frame identifier as the index key and pointing to the candidate record; Deduplication of duplicate acoustic wave candidate frames is performed based on the frame identifier index; If the candidate screen projection code is the same as the service screen projection code, the candidate network address and the current discovery address both belong to the current local network segment and there is no identity conflict, delete the network address index entry with the current binding address before the update as the index key, create a network address index entry with the current discovery address as the index key, and update the current binding address in the candidate record to the current discovery address.

12. The candidate management device for projection devices according to claim 10, characterized in that, It also includes a candidate protection module, which is configured to detect at least one of the following abnormal conditions: the candidate record expires, the network fingerprint does not match, the candidate projection code is different from the service projection code, or the candidate network address establishes a unique correspondence with another device discovery record. If the abnormal conditions are detected, the candidate protection module outputs a prohibition association command to the candidate association module and the connection parameter module to remove the association between the device discovery record identifier and the candidate record identifier, and to prevent the generation of the connection parameter set used to establish the screen projection connection.

13. A screen mirroring initiation electronic device, characterized in that, include: An audio input interface for receiving sound wave signals captured by a microphone; The network communication interface is used to receive network service discovery data and send screen mirroring connection data. The memory is used to store computer programs, candidate records, projection code indexes, and network address indexes. as well as The processor is connected to the audio input interface, the network communication interface, and the memory, respectively. When the processor executes the computer program, it implements the screen projection device candidate management method according to any one of claims 1 to 9.

14. A computer program product, characterized in that, Including computer programs, When the computer program runs on the processor, it causes the processor to implement the screen projection device candidate management method according to any one of claims 1 to 9.

Images