Equipment control system, method and equipment

Abstract

The embodiment of the invention discloses a device control system, method and device, and is suitable for the technical field of computers, the device control system comprises a first device and a second device, and the first device and the second device are connected; the first device responds to a first operation that a user triggers a foreground to run a first application, and owner recognition processing is executed; under the condition that the first device recognizes that the user is not the owner, a first interface is displayed on the second device, the first interface comprises a first window, the first window comprises first prompt information and a first control, and the first prompt information reminds that the first application is being used by a non-owner; and under the condition that the second equipment receives the operation of the user on the first control, controlling the first equipment to lock the screen. According to the embodiment of the invention, when the terminal equipment is used by a non-owner, the use permission of the non-owner can be limited.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a device control system, method, and device. Background Art

[0002] Mobile devices, especially mobile phones, have gradually become the center of our digital worlds, carrying vast amounts of personal information and private data. While their rich functionality has brought significant convenience to their owners, it has also created numerous challenges.

[0003] In actual applications, it is found that when a terminal device is used by someone other than the owner, that is, a non-owner, the non-owner may use applications in the terminal device that the owner does not want others to use without the owner's permission, which may easily cause unnecessary losses or troubles to the owner of the terminal device.

[0004] Therefore, when a terminal device is used by someone other than the owner, the non-owner's usage rights need to be restricted. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a device control system, method, and device that can restrict the usage rights of non-owners when a terminal device is used by non-owners.

[0006] In the first aspect, an embodiment of the present application provides a device control system, which may include a first device and a second device, wherein a connection has been established between the first device and the second device. First, the first device may perform an owner identification process in response to a first operation of a user on the first device. The owner identification process may identify whether the user is the owner, and the first operation may trigger the running of a first application in the foreground. Afterwards, when the first device identifies that the user is not the owner, the second device may display a first interface. The first interface includes a first window, and the first window includes a first prompt message and a first control. The first prompt message may remind the first device that the first application is being used by a non-owner. Finally, when the second device receives an operation by the user on the first control, the second device may control the first device to lock the screen.

[0007] Optionally, the first device may be a mobile phone and the second device may be a wearable device. Since the owner usually wears a wearable device, when the first device is a mobile phone and the second device is a wearable device, the owner can receive application usage reminders in a timely manner.

[0008] Optionally, the wearable device may be a watch.

[0009] The user on the first device may be the device owner or a non-owner, while the user on the second device is typically the device owner. For ease of description, the user on the first device may be referred to as the first user, and the user on the second device may be referred to as the second user. It is understood that the first user and the second user may or may not be the same user.

[0010] The first application is typically a sensitive application. In other words, the first application is typically an application from a preset list of applications. The first device only uses sensitive applications and issues reminders to the user via the second device, thereby avoiding unnecessary loss of computing resources.

[0011] Optionally, the first operation may include a start operation and a switch operation. The start operation is used to trigger the start of the first application, and the switch operation is used to trigger the first application to be switched to the foreground for execution.

[0012] In an embodiment of the present application, the first device can promptly notify the owner of an attempt by a non-owner to use the first application on the first device through the second device. This allows the owner to promptly learn of the non-owner's use of the first application on the first device without having to stare at the first device. In addition, the second device can respond to the owner clicking the first control (also known as a one-click lock control or lock control) on the second device and promptly control the first device to lock the screen, thereby preventing the non-owner from using the first application on the first device. In other words, the owner can conveniently and flexibly restrict the non-owner's use of the first application on the first device through the second device.

[0013] In a first possible implementation manner of the first aspect, the information transmission channel between the first device and the second device may include a first channel and a second channel.

[0014] The first channel is the information transmission channel between the application on the first device and the second device. The second channel is the information transmission channel between the system service on the first device and the second device. Applications are located in the application layer of the Android system, while system services are located in the application framework layer of the Android system. In other words, the first channel and the second channel are two different types of channels.

[0015] In practice, the second channel is an information transmission channel corresponding to system services, used to transmit system service information. The first channel is an information transmission channel corresponding to applications, used to transmit application information.

[0016] In practice, the first channel is an information transmission channel between a target application in the first device and the second device. The target application is typically an application used to connect the first device and the second device. In practice, the target application is typically a sports and health application.

[0017] In the embodiment of the present application, two information transmission channels are established between the first device and the second device to meet different data transmission requirements. In addition, data transmission through multiple channels at the same time can improve data transmission efficiency.

[0018] It should be noted that the first channel and the second channel are usually two channels established when the first device is connected to the second device.

[0019] Optionally, the first channel and the second channel may be based on the same transmission protocol for information transmission. For example, the first channel and the second channel may be based on the Bluetooth transmission protocol for information transmission.

[0020] Optionally, the first channel and the second channel may be based on different transmission protocols for information transmission. For example, the first channel is based on the Bluetooth transmission protocol for information transmission, and the second channel is based on the WLAN transmission protocol for information transmission.

[0021] It is understandable that the embodiment of the present application does not specifically limit the transmission protocols used by the first channel and the second channel.

[0022] In a second possible implementation of the first aspect, upon recognizing that the user is not the device owner, the first device may send a notification message to the second device. The notification message indicates that the first application is being used by a non-device owner. The second device may then receive the notification message and display the first interface based on the notification message.

[0023] In the embodiment of the present application, the first device can send a notification message to the second device to notify the second device to promptly display the corresponding information, that is, the first interface, to the device owner. In this way, the device owner can promptly know through the second device that the first application on the first device is being used, and can choose whether to prevent non-users from using the first application.

[0024] In some optional embodiments of the present application, the first device generally sends notification information to the second device through the second channel.

[0025] In the embodiment of the present application, since the second channel is the information transmission channel corresponding to the system service, and since the system service is usually always in a live state, that is, the second channel can always be turned on, therefore, in the embodiment of the present application, the first device sends notification information to the second device through the second channel, which can ensure that when the first device detects that the first application is being used, it can promptly prompt the owner of the device through the second device.

[0026] In a third possible implementation of the first aspect, the second device may send a lock instruction to the first device in response to a user operation on the first control. The first device may then receive the lock instruction, respond to the lock instruction, and lock the electronic screen of the first device.

[0027] In an embodiment of the present application, the second device can instruct the first device to lock the screen in time by sending a lock instruction to the first device, thereby preventing non-owners from using the above-mentioned first application in time according to the owner's intention.

[0028] In some optional embodiments of the present application, the second device usually sends a locking instruction to the first device through the second channel.

[0029] In the embodiment of the present application, since the second channel is the information transmission channel corresponding to the system service, that is, the second channel can always be in the open state, therefore, the second device sends a lock instruction to the first device through the second channel, which can ensure that the second device transmits the lock instruction to the first device in a timely manner, so that the first device can lock the screen in time.

[0030] In some optional embodiments of the present application, the second channel is an encrypted transmission channel.

[0031] In the embodiment of the present application, the second channel is an encrypted transmission channel, which can ensure the security of information transmission between the first device and the second device.

[0032] In some optional embodiments of the present application, the first device includes a first collaborative service module, and the second device includes a second collaborative service module. The first device sends notification information to the second device, typically by the first collaborative service module sending the notification message to the second collaborative service module.

[0033] In some optional embodiments of the present application, the second device sends a locking instruction to the first device, typically, the second collaborative service module sends the locking instruction to the first collaborative service module.

[0034] Among them, the first collaborative service module is a module in the first device for implementing application usage reminders. The second collaborative module is a module in the second device for implementing application usage reminders. In practice, the first collaborative service module in the first device is used in conjunction with the second collaborative service module in the second device, so that when a non-owner uses a sensitive application on the first device, the second device can promptly display the first window to the owner to remind the owner that the sensitive application (for example, the first application) is being used by the non-owner. In this way, the owner can be aware in a timely manner that the sensitive application is being used by the non-owner, and the owner can operate the controls on the first window based on the actual situation to choose to stop or not stop the non-owner from using the current sensitive application.

[0035] In some optional embodiments of the present application, the first collaborative service module and the second collaborative service module usually transmit information through the second channel, so as to ensure timely transmission of information.

[0036] In a fourth possible implementation of the first aspect, a target application is installed in the first device, and the target application includes a first setting interface (or called a sensitive application setting interface), and the first setting interface may include a first switch control (or called a sensitive application usage reminder switch).

[0037] In some optional embodiments of the present application, the first device may enable an application usage reminder function in response to a user's operation of turning on a first switch control.

[0038] In the embodiment of the present application, since non-device owners typically do not enable the application usage reminder function on the first device, the first device directly enables the application usage reminder function in response to the user's operation of turning on the first switch control (or referred to as a function activation operation), thereby quickly enabling the application usage reminder function. It is understandable that only when the application usage reminder function is enabled will the first device promptly send an application usage reminder to the user through the second device when the non-device owner uses a sensitive application.

[0039] In some optional embodiments of the present application, the first device may perform owner identification processing in response to the user turning off the first switch control, and turn off the application usage reminder function when the user is identified as the owner.

[0040] In an embodiment of the present application, when the first device responds to the user's operation of closing the first switch control (or called a function closing operation), it first performs owner identification. Only when the owner is identified, the application usage reminder function is turned off. This can avoid the situation where the owner is unable to promptly know that a non-owner is using sensitive applications due to the non-owner deliberately turning off the application usage reminder function.

[0041] In some optional embodiments of the present application, the first device performs the owner identification process in response to the user triggering the first operation of running the first application in the foreground. Specifically, it can be implemented as follows: when the application usage reminder function is turned on, the first device performs the owner identification process in response to the user triggering the first operation of running the first application in the foreground.

[0042] In an embodiment of the present application, the first device will respond to the first operation and execute subsequent application usage reminders only when the application usage reminder function is turned on, thereby avoiding unnecessary loss of computing resources.

[0043] In a fifth possible implementation of the first aspect, the first setting interface (or referred to as the sensitive application setting interface) of the target application may include a preset application list, wherein the first application belongs to the preset application list.

[0044] It is understandable that the first device may set the first application as a sensitive application before the non-owner uses the first application, that is, add the first application to the preset application list.

[0045] The applications in the preset application list are sensitive applications. By default, the preset application list may be empty, that is, there are no applications in the preset application list. It is understandable that only when the first application is a sensitive application will the first device respond to the first operation on the first application and perform the sensitive application usage reminder, thereby avoiding unnecessary computing resource consumption.

[0046] Optionally, the preset application list can be implemented as a list.

[0047] In some optional embodiments of the present application, the first device can perform owner identification processing in response to a user's deletion operation on an application in a preset application list (or called an application deletion operation), and when the user is identified as the owner, the application targeted by the deletion operation will be deleted from the preset application list.

[0048] In an embodiment of the present application, when the first device responds to the user's application deletion operation, it first performs owner identification. Only when the owner is identified, the application involved in the application deletion operation is deleted from the preset application list. This can avoid the situation where the owner is unable to promptly know that the non-owner is using sensitive applications due to the non-owner deliberately deleting the sensitive applications.

[0049] In a sixth possible implementation manner of the first aspect, the first setting interface of the target application (or referred to as the sensitive application setting interface) may further include a setting control (or referred to as a management application control).

[0050] The first device may perform owner identification processing in response to a user operation on a setting control (or an application adjustment operation), and display a second interface (or an application management interface) when the user is identified as the owner. The second interface may include multiple second switch controls (or application switches). Each second switch control may correspond to an application.

[0051] Optionally, in response to the user's operation of turning on the second switch control, the first device may add an application corresponding to the operated second switch control to a preset application list.

[0052] Optionally, in response to the user's operation of closing the second switch control, the first device may delete the application corresponding to the operated second switch control from the preset application list.

[0053] In an embodiment of the present application, the first device displays second switch controls corresponding to multiple applications on the second interface, which can facilitate the user to set multiple applications as sensitive applications at the same time, or to delete multiple applications from the preset application list at the same time, that is, to delete multiple sensitive applications at the same time, which helps to improve the user experience.

[0054] In addition, since the non-owner may turn on one or more second switch controls at the same time on the second interface, may also turn off one or more second switch controls at the same time, and may also turn on some second switch controls and turn off other second switch controls, in order to prevent the non-owner from deleting sensitive applications, the first device will only enter the second interface when it identifies the owner. This can avoid the situation where the owner is unable to promptly know that the non-owner is using sensitive applications due to the non-owner deliberately deleting sensitive applications.

[0055] In some optional embodiments of the present application, when a sensitive application is added to the first device for the first time, the first setting interface of the target application (or referred to as the sensitive application setting interface) may further include an application addition control.

[0056] In practice, the first device may display an add application interface in response to a user operation for adding an application control (or referred to as an application adding operation).

[0057] Optionally, in response to the user turning on the second switch control on the add application interface, the first device may add the application corresponding to the operated second switch control to the preset application list.

[0058] In an embodiment of the present application, since non-owners usually do not add sensitive applications to the first device, when responding to the target user's application addition operation, the first device directly adds the application involved in the application addition operation as a sensitive application, which can achieve rapid addition of sensitive applications.

[0059] In a seventh possible implementation manner of the first aspect, the first setting interface of the target application (or referred to as the sensitive application setting interface) may further include a record viewing control (or referred to as a sensitive application usage record control).

[0060] In this implementation, the first device may also display historical access records of non-owners accessing applications in a preset application list in response to a user operation on a record viewing control (or referred to as a record viewing operation).

[0061] The above record viewing operation is generally used to view the usage records of sensitive applications. Sensitive applications are applications in a preset application list.

[0062] In an embodiment of the present application, the first device can respond to the user's record viewing operation and display the sensitive application usage record to the user, which can facilitate the user to view the sensitive application usage record on the first device as a whole, helping to improve the user experience.

[0063] It should be noted that the aforementioned app deletion, app addition, app adjustment, feature activation, feature deactivation, and history review operations all operate on the target app's first settings interface (or sensitive app settings interface). This means that users can perform operations within the same interface within the same app to add or delete sensitive apps, enable or disable the app usage reminder function (or sensitive app usage reminder function), and view historical access records of sensitive apps by non-users (or sensitive app usage records). This facilitates user operations and helps improve the user experience.

[0064] In an eighth possible implementation manner of the first aspect, the target application is a sports and health application.

[0065] In the embodiments of the present application, since the first device and the second device are typically paired and connected via a sports and health application, when the target application is a sports and health application, operations such as adding an application, deleting an application, disabling a function, and viewing a record are all performed on the interface of the sports and health application. This allows for the setting of sensitive applications, the setting of application usage reminders, and the viewing of sensitive application usage records with minimal changes to the device framework, thereby facilitating user operations and reducing development costs.

[0066] In some optional embodiments of the present application, the first device is used to perform owner identification processing in response to the user's first operation, which can be specifically implemented as follows: when the application usage reminder function is turned on and the first application belongs to the preset application list, the first device performs owner identification processing in response to the user's first operation.

[0067] In an embodiment of the present application, the first device will only respond to the first operation and execute subsequent application usage reminders when the application usage reminder function is turned on and the first application corresponding to the first operation triggered by the user is a sensitive application, thereby avoiding unnecessary loss of computing resources.

[0068] In a ninth possible implementation of the first aspect, the first window displayed by the second device may further include a second control (or referred to as a no-remind control). When the second device receives a user operation on the second control, the first device may run the first application in the foreground.

[0069] In the embodiment of the present application, when the second control of the second device is triggered, it indicates that the owner allows non-owners to use the first application. At this time, the first device can run the first application in the foreground.

[0070] It is understandable that in some application scenarios, the first window displayed by the second device may not have the above-mentioned second control. In this application scenario, when the owner clicks the first control on the first window, the second device believes that the owner does not allow non-owners to use the above-mentioned first application. And when the owner does not click the first control on the first window, the second device believes that the owner allows non-owners to use the above-mentioned first application. If the owner does not allow non-owners to use the first application, the first device can execute a lock screen, that is, the first device cannot run the above-mentioned first application in the foreground. If the owner allows non-owners to use the first application, the first device will not execute a lock screen, that is, the first device can run the above-mentioned first application in the foreground.

[0071] In some optional embodiments of the present application, when the second device receives a user operation on the second control, the first device can also continue to run the first application in the foreground within a preset effective time period when it receives the first operation on the first application again.

[0072] The preset effective duration is usually a pre-set duration, for example, 30 minutes.

[0073] In an embodiment of the present application, if the owner allows non-owners to use the above-mentioned first application, the first device may no longer send usage reminders to the owner for the first application within a period of time. This can not only save unnecessary computing resource consumption, but also avoid frequent reminders causing excessive interference to the owner, which helps to improve the user experience.

[0074] In some optional embodiments of the present application, after the second device displays the first interface, the second device may, in response to the user not operating the controls on the first window within a first time period, initiate an operation reminder. For example, the second device may vibrate or continuously vibrate for a certain period of time, such as 5 seconds.

[0075] In some optional embodiments of the present application, after the second device displays the first interface, the second device may also stop displaying the first window in response to the user not operating the controls on the first window within the first time period.

[0076] In a tenth possible implementation of the first aspect, the process of the first device performing the owner identification process may include: first, the first device may collect user data. Then, the first device may determine whether the user is the owner based on the user data and a pre-recorded owner template.

[0077] The user data includes at least one of the following: a face image to be tested, breathing audio data to be tested, IMU data to be tested, and touch screen data to be tested.

[0078] Among them, the owner template includes at least one of the following: a face template, a breathing sound feature template, and a cross-modal feature template. The cross-modal feature template is a comprehensive feature of the owner under multiple modalities.

[0079] In an embodiment of the present application, the first device can identify whether the user is the owner of the device based on one or more of the user's corresponding facial data, breathing audio data, IMU data, and touch screen data. The owner identification can be realized using data in multiple modalities, and the advantages of different modal data can complement each other, thereby improving the stability and accuracy of the owner identification.

[0080] In an eleventh possible implementation manner of the first aspect, the first device determines whether the user is the owner based on the user data and a pre-recorded owner template. The specific process may be:

[0081] First, when there is a face in the face image to be tested, the first device can perform owner recognition based on the face image to be tested and the face template to obtain a face recognition score.

[0082] Then, the first device may extract the breathing sound features of the user to be tested from the breathing audio data to be tested when the face recognition score is greater than a preset face recognition lower threshold and less than a preset face recognition upper threshold.

[0083] Finally, the first device can input the breathing sound features to be tested, the IMU data to be tested, the touch screen data to be tested and the face recognition score into the multimodal fusion owner recognition model to obtain the owner indication information, and the owner indication information indicates whether the user is the owner.

[0084] Optionally, the face recognition upper threshold may be a FAR threshold for face recognition, and the face recognition lower threshold may be a FRR threshold for face recognition.

[0085] In practice, a facial recognition score between the lower and upper facial recognition thresholds typically indicates a poor facial recognition result. That is, when facial recognition is not effective, the first device can combine multimodal data to further accurately identify the owner, thereby improving the accuracy of owner identification. Furthermore, because facial recognition accuracy is already sufficiently high most of the time, the use of multimodal data for owner identification is relatively rare. This allows the first device to conserve computational resources and reduce power consumption while maintaining the accuracy of owner identification, thereby further enhancing the user experience.

[0086] In a twelfth possible implementation manner of the first aspect, in the multimodal fusion machine owner identification model, the data processing process may specifically be:

[0087] First, the IMU data to be tested, the touch screen data to be tested, and the respiratory sound features to be tested are encoded by a cross-modal encoder to obtain the cross-modal features to be tested.

[0088] Then, the cross-modal similarity between the cross-modal feature to be measured and the cross-modal feature template in the machine master template is determined, and the respiratory sound similarity between the respiratory sound feature to be measured and the respiratory sound feature template in the machine master template is determined.

[0089] Afterwards, the multimodal recognition score is determined based on the cross-modal similarity, breathing sound similarity, and face recognition score. The multimodal recognition score is the comprehensive recognition result of the user in multiple modalities.

[0090] Finally, based on the multimodal recognition score and the pre-set fusion threshold, it is determined whether the user is the owner of the device.

[0091] Among them, the cross-modal encoder can encode data of multiple modalities to obtain comprehensive features under multiple modalities, that is, cross-modal features.

[0092] In an embodiment of the present application, the multimodal fusion machine owner recognition model uses a cross-modal encoder to fuse IMU data, touch screen data, and breathing sound features at the feature level. The cross-modal similarity (or cross-modal recognition score), breathing sound similarity (or breathing recognition score), and face recognition score are then fused at the decision level to obtain a multimodal recognition score. Among them, stronger modalities, such as face modality, are not involved in feature-level fusion. This can reduce the coupling between data and effectively utilize the complementary advantages between multiple modalities to improve the overall recognition success rate of the model.

[0093] In a second aspect, embodiments of the present application provide a device control method that can be applied to a first device. In response to a user triggering a first operation to bring a first application into the foreground, the first device can perform owner identification. Subsequently, upon recognizing that the user is not the device owner, the first device can send a notification to a second device indicating that the first application is being used by a non-device owner. Finally, the first device can lock the electronic screen in response to receiving a lock instruction from the second device in response to the notification.

[0094] As an embodiment of the present application, the first device may be a mobile phone, and the second device may be a wearable device. Optionally, the wearable device may be a watch.

[0095] As an embodiment of the present application, an information transmission channel between a first device and a second device may include a first channel and a second channel. The first channel is an information transmission channel between an application in the first device and the second device, and the second channel is an information transmission channel between a system service in the first device and the second device.

[0096] Among them, applications are located in the application layer of the Android system, and system services are located in the application framework layer of the Android system.

[0097] As an embodiment of the present application, the first device may also send notification information to the second device when it is identified that the user is not the device owner, where the notification information indicates that the first application is being used by a non-device owner.

[0098] As an embodiment of the present application, the first device sending notification information to the second device can be implemented as follows: the first device sends the notification information to the second device through the second channel.

[0099] As an embodiment of the present application, locking the electronic screen of the first device may be implemented as follows: the first device locks the electronic screen of the first device in response to receiving a locking instruction.

[0100] As an embodiment of the present application, the second channel is an encrypted transmission channel.

[0101] As an embodiment of the present application, a target application is installed in the first device, the target application includes a first setting interface, and the first setting interface may include a first switch control.

[0102] Specifically, the first device may enable the application usage reminder function in response to the user turning on the first switch control. The first device may also perform owner identification processing in response to the user turning off the first switch control, and disable the application usage reminder function when the user is identified as the owner.

[0103] Among them, the first device performs the owner identification processing in response to the first operation triggered by the user to run the first application in the foreground. Specifically, it can be implemented as follows: when the application usage reminder function is turned on, the first device performs the owner identification processing in response to the first operation triggered by the user to run the first application in the foreground.

[0104] As one embodiment of the present application, the first settings interface of the target application may include a list of preset applications, and the first application belongs to the preset application list. In response to a user's deletion operation on an application in the preset application list, the first device may perform owner identification processing and, upon identifying the user as the owner, delete the application targeted by the deletion operation from the preset application list.

[0105] As one embodiment of the present application, a first settings interface of a target application may include a settings control. In response to a user's operation on the settings control, the first device may perform owner identification processing. Upon identifying the user as the owner, the first device may display a second interface including multiple second switch controls. Each second switch control corresponds to an application.

[0106] In response to the user turning on the second switch control, the first device may add the application corresponding to the operated second switch control to the preset application list. In response to the user turning off the second switch control, the first device may delete the application corresponding to the operated second switch control from the preset application list.

[0107] As an embodiment of the present application, the first setting interface of the target application may further include a record viewing control. In response to the user's operation on the record viewing control, the first device may display the historical access records of the non-owner accessing the applications in the preset application list.

[0108] As an embodiment of the present application, the target application is a sports and health application.

[0109] As an embodiment of the present application, the first device performs the owner identification process in response to the user's first operation, which can be implemented as follows: when the application usage reminder function is turned on and the first application belongs to the preset application list, the first device performs the owner identification process in response to the user's first operation.

[0110] As an embodiment of the present application, the first window further includes a second control; when the second device receives a user operation on the second control, the first device can run the first application in the foreground.

[0111] As an embodiment of the present application, when the second device receives a user operation on the second control, the first device may also run the first application in the foreground if it receives the first operation on the first application again within a preset effective time.

[0112] As an embodiment of the present application, the first device performs owner identification processing, which can be implemented as follows: First, the first device can collect user data. The user data includes at least one of the following: a face image to be tested, breathing audio data to be tested, IMU data to be tested, and touch screen data to be tested. Then, based on the user data and the pre-recorded owner template, it is determined whether the user is the owner of the device. The owner template includes at least one of the following: a face template, a breathing sound feature template, and a cross-modal feature template. The cross-modal feature template is a comprehensive feature of the owner under multiple modalities.

[0113] As an embodiment of the present application, the first device determines whether the user is the owner of the device based on the user data and the pre-entered owner template. Specifically, it can be implemented as follows: First, the first device can identify the owner of the device based on the face image to be tested and the face template when there is a face in the face image to be tested, and obtain a face recognition score. Afterwards, the first device can extract the breathing sound features of the user to be tested from the breathing audio data to be tested when the face recognition score is greater than a preset face recognition lower limit threshold and less than a preset face recognition upper limit threshold. Finally, the first device can input the breathing sound features to be tested, the IMU data to be tested, the touch screen data to be tested and the face recognition score into the multimodal fusion owner recognition model to obtain owner indication information, and the owner indication information indicates whether the user is the owner of the device.

[0114] As an embodiment of the present application, in the multimodal fusion machine owner identification model, the data processing process can be as follows: first, the IMU data to be tested, the touch screen data to be tested, and the breathing sound features to be tested are encoded by a cross-modal encoder to obtain the cross-modal features to be tested; then, the cross-modal similarity between the cross-modal features to be tested and the cross-modal feature template in the machine owner template is determined, and the breathing sound similarity between the breathing sound features to be tested and the breathing sound feature template in the machine owner template is determined; thereafter, the multimodal recognition score is determined based on the cross-modal similarity, the breathing sound similarity, and the face recognition score, wherein the multimodal recognition score is the comprehensive recognition result of the user under multiple modalities; finally, based on the multimodal recognition score and a pre-set fusion threshold, it is determined whether the user is the machine owner.

[0115] As an embodiment of the present application, the device control method can implement any function of the first device in the first aspect described above.

[0116] In a third aspect, an embodiment of the present application provides a device control method that can be applied to a second device, wherein the second device can receive notification information from a first device. The notification information indicates that a first application on the first device is being used by a non-device owner. Thereafter, in response to the notification information, the second device can display a first interface, wherein the first interface includes a first window, the first window includes a first prompt information and a first control, and the first prompt information reminds that the first application is being used by a non-device owner. Finally, the second device can send a lock instruction to the first device in response to the user's operation on the first control, and the lock instruction instructs the first device to lock the screen.

[0117] As an embodiment of the present application, the first device is a mobile phone, and the second device is a wearable device. Optionally, the wearable device can be a watch.

[0118] As an embodiment of the present application, the second device displays the first interface, which can be implemented as follows: the second device displays the first interface in response to receiving the notification information.

[0119] As an embodiment of the present application, the information transmission channel between the first device and the second device includes a second channel, and the second channel is an information transmission channel between the system service in the first device and the second device.

[0120] As an embodiment of the present application, the second device may send a lock instruction to the first device in response to a user operation on the first control.

[0121] As an embodiment of the present application, the second device sends a locking instruction to the first device, which can be implemented as follows: the second device sends the locking instruction to the first device through the second channel.

[0122] As an embodiment of the present application, the second channel is an encrypted transmission channel.

[0123] As an embodiment of the present application, the information transmission channel between the first device and the second device further includes a first channel, which is an information transmission channel between a target application in the first device and the second device.

[0124] As an embodiment of the present application, the first window also includes a second control. When the second control is triggered, it instructs the owner to allow the first device to run the first application in the foreground.

[0125] As an embodiment of the present application, the device control method can implement any function of the first device in the first aspect described above.

[0126] In a fourth aspect, embodiments of the present application provide a terminal device, which is the first device described in the first or second aspect above. The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the functions of the first device described in the first or second aspect above are realized.

[0127] In a fifth aspect, embodiments of the present application provide a wearable device, which is the second device in the first or third aspect. The wearable device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the functions of the first device in the first or third aspect.

[0128] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the second or third aspects above.

[0129] In a seventh aspect, embodiments of the present application provide a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the method according to any one of the second or third aspects above. The chip system may be a single chip or a chip module composed of multiple chips.

[0130] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] Figure 1 A schematic diagram of a scenario for providing usage reminders for sensitive applications provided in an embodiment of the present application;

[0132] Figure 2 The architecture of the system provided in the embodiments of the present application;

[0133] Figure 3A This is a software structure diagram of the terminal device according to an embodiment of the present application;

[0134] Figure 3B A software structure diagram of a wearable device according to an embodiment of the present application;

[0135] Figure 4 A schematic diagram of a process for establishing a connection between a mobile phone and a watch provided in an embodiment of the present application;

[0136] Figure 5A A schematic diagram of the process of adding a sensitive application to a mobile phone for the first time provided in an embodiment of the present application;

[0137] Figure 5B A schematic diagram of a process for deleting sensitive applications on a mobile phone provided in an embodiment of the present application;

[0138] Figure 5C A schematic diagram of another process of deleting sensitive applications on a mobile phone provided in an embodiment of the present application;

[0139] Figure 5D A schematic diagram of the process of disabling the reminder function for sensitive application usage provided in an embodiment of the present application;

[0140] Figure 5E A schematic diagram of the process of viewing sensitive application usage records provided in an embodiment of the present application;

[0141] Figure 6A A schematic diagram of the process of identifying the owner of the device according to an embodiment of the present application;

[0142] Figure 6B A schematic diagram of the process of identifying the owner of the device based on breathing sounds provided in an embodiment of the present application;

[0143] Figure 6C A schematic diagram of the process of performing machine owner identification based on a multimodal fusion machine owner identification model provided in an embodiment of the present application;

[0144] Figure 7 A timing diagram showing the collaborative operation of a mobile phone and a watch provided in an embodiment of the present application;

[0145] Figure 8A A schematic diagram of a first interface displayed in a watch provided in an embodiment of the present application;

[0146] Figure 8B A schematic diagram of an interactive scenario of a device control method provided in an embodiment of the present application;

[0147] Figure 9A A schematic diagram of the structure of a device control system provided in an embodiment of the present application is shown;

[0148] Figure 9B A schematic structural diagram of a first device provided in an embodiment of the present application is shown;

[0149] Figure 9C A schematic structural diagram of a second device provided in an embodiment of the present application is shown;

[0150] Figure 10A A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0151] Figure 10B A schematic diagram of the structure of a wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0152] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0153] The following describes some concepts that may be involved in the embodiments of this application:

[0154] (1) Multiple: Unless otherwise specified, in the embodiments of the present application, multiple refers to two or more.

[0155] (2) Inertial Measurement Unit (IMU): In the embodiments of the present application, an IMU is a sensor used to measure and track the posture (such as direction, angle, and velocity) of an object. The IMU contains multiple measurement units, including accelerometers, gyroscopes, and magnetometers. Among them, the accelerometer can measure the linear acceleration of an object along the x, y, and z axes; the gyroscope can measure the angular velocity of an object around the x, y, and z axes; and the magnetometer can measure the magnetic field strength and direction of an object in the Earth's magnetic field.

[0156] In the embodiment of the present application, the IMU data may include the acceleration, pitch angle, roll angle, etc. of the terminal device in various directions. The IMU data is the data collected by the IMU in the terminal device.

[0157] (3) Multimodality: In the embodiments of this application, multimodality generally refers to multiple types. Multimodal data generally refers to multiple different types of data. For example, facial data, breathing audio data, touch screen data, and IMU data are four different types of data.

[0158] (4) Device owner template, user data: In the embodiment of the present application, the device owner template is usually pre-entered or stored information used to describe the characteristics of the device owner. The device owner template may include a face template, a respiratory sound feature template, and a cross-modal feature template. Among them, the face template is information pre-entered or stored for describing the facial features of the device owner, for example, it can be an image of the device owner's face. The respiratory sound feature template is information pre-entered or stored for describing the respiratory sound characteristics of the device owner. The cross-modal feature template is information pre-entered or stored for describing the comprehensive characteristics of the device owner in multiple modalities.

[0159] In practice, a mobile phone can have a single owner template or multiple owner templates. In other words, a mobile phone can have one or more owner templates. An owner template typically corresponds to a single owner. In some application scenarios, an owner template can contain one or more facial templates, one or more breathing sound feature templates, and one or more cross-modal feature templates.

[0160] In an embodiment of the present application, the user data may include at least one of the following: a facial image to be tested, breathing audio data to be tested, IMU data to be tested, and touch screen data to be tested. The facial image to be tested is a facial image of the user to be tested. The breathing audio data to be tested is breathing audio data of the user to be tested. The IMU data to be tested is IMU data of the user to be tested. The touch screen data to be tested is touch screen data of the user to be tested.

[0161] (5) False Acceptance Rate (FAR), False Rejection Rate (FRR): In the embodiment of the present application, FAR refers to the proportion of samples that are incorrectly classified as other categories in the classification problem. In the field of machine learning and pattern recognition, we usually focus on two types of errors: false positives and false negatives. False positives refer to the situation where a negative sample is mistakenly classified as a positive sample, while false negatives refer to the situation where a positive sample is mistakenly classified as a negative sample. The false recognition rate can be calculated by the following formula: false recognition rate = (number of false positive samples + number of false negative samples) / total number of samples.

[0162] In practice, the lower the false positive rate, the better the classifier performance. In real applications, we usually weigh the weights of false positives and false negatives based on specific needs, choose the appropriate classifier or adjust the model parameters to achieve the optimal false positive rate.

[0163] In the embodiment of the present application, FRR refers to the probability of judging a sample that should have been matched successfully as a matching failure during the category matching process.

[0164] (6) Sensitive applications. In the embodiment of the present application, sensitive applications generally refer to applications that the owner does not want others to use.

[0165] In practice, device owners can manage sensitive apps using a pre-set app list. For example, they can add one or more apps installed on their device to this pre-set app list. Apps added to this pre-set app list are considered sensitive apps. Subsequently, when an app on this pre-set app list is used by someone other than the device owner, the device can promptly send a usage reminder to the owner's wearable device.

[0166] Mobile devices, especially mobile phones, have gradually become the center of our digital worlds, carrying vast amounts of personal information and private data. While their rich functionality has brought significant convenience to their owners, it has also created numerous challenges.

[0167] The following uses a mobile phone as an example to illustrate several usage scenarios in which the terminal device may cause trouble to the owner. It is understandable that the terminal device in the embodiments of the present application is not limited to a mobile phone, but may also be other devices, such as a tablet, a laptop, etc.

[0168] Usage scenario one: the owner lends the phone to the child to look up information, such as writing materials or problem-solving ideas, and the child ends up watching videos or playing games on the phone.

[0169] For example, the owner of the phone (for example, the child's mother) unlocks the phone and lends it to the child to check writing materials. After the owner leaves, the child uses the phone to watch videos.

[0170] Usage scenario two: After the child completes a designated task, the owner rewards the child with a game session. As a result, the child uses the phone to recharge and buy gaming equipment.

[0171] For example, after a child finished two exam papers, the owner rewarded the child with a half-hour game session. After unlocking the phone, the owner gave the child the phone. However, after the owner left, the child used the phone to buy game skins.

[0172] Through the two exemplary usage scenarios listed above, we can see that when a terminal device is used by someone other than the device owner (or a non-owner), the non-owner may, without the device owner's permission, use applications on the terminal device that the device owner does not want others to use, which can easily cause unnecessary losses or trouble to the device owner. In this application, for ease of description, applications that the device owner does not want others to use are referred to as sensitive applications.

[0173] It should be noted that a user using an application on a terminal device generally refers to the terminal device responding to a first operation of the user to start the application or switch the application from background operation to foreground operation, wherein the first operation is an operation for triggering the application to run in the foreground.

[0174] Based on the above description, when a terminal device is used by a non-owner, the non-owner's usage rights need to be restricted, or in other words, the non-owner's use of sensitive applications on the terminal device needs to be restricted.

[0175] In view of the above technical problems existing in the related technologies, in order to restrict non-users from using sensitive applications in the terminal device, in the embodiment of the present application, taking the first device as a mobile phone and the second device as a watch as an example, Figure 1 , the mobile phone can perform owner recognition in response to the user clicking the application icon 101 of the WeChat application. Afterwards, if the owner recognition fails, that is, if the owner is not recognized, the mobile phone will display the message "Owner recognition failed" 102 on the mobile phone. Then, the watch can display the first window 103. The first window 103 may include the message "Your Magic V2 is accessing WeChat" 104, and may also include a one-key lock control 105 and a no more reminder control 106. Among them, Figure 1 A schematic diagram of a scenario for providing usage reminders for sensitive applications provided in an embodiment of the present application.

[0176] like Figure 1 As shown, if a non-owner (e.g., a child) uses WeChat, a sensitive app on their phone, the phone can send a reminder to the watch in real time. This way, the owner (e.g., the child's mother) wearing the watch can be notified of the non-owner's use of sensitive apps on their phone through the watch.

[0177] Combine Figure 1 The owner can also further operate on the watch to restrict non-owners (such as children) from using sensitive applications on the phone. Figure 1 In the example, the watch can respond to the owner clicking the one-touch lock control 105 on the first window 103 to notify the phone to lock the screen. When the phone is locked, non-owners cannot use sensitive applications, such as WeChat. For another example, the watch can respond to the owner clicking the no longer remind control 106 on the first window 103 to notify the phone that the current owner has tacitly allowed non-owners to use sensitive applications. At this time, the phone can run sensitive applications, such as WeChat, in the foreground.

[0178] Combine Figure 1 As you can see, the phone can use the watch to promptly notify the owner of any attempts by non-owners to access sensitive apps on the phone. This means that even when the phone is being used by someone other than the owner, the owner can still be informed of their use of sensitive apps without having to physically monitor the phone. Furthermore, the watch can respond to the owner's taps on the watch controls, notifying the phone to perform the corresponding action, such as locking or unlocking the screen. This means that the owner can use the watch to conveniently and flexibly restrict non-owners' access to sensitive apps on their phone.

[0179] Understandably, in some scenarios, the owner may not click on the controls in first window 103. In this case, the watch or phone may assume that the owner has granted permission to non-owners to use the current sensitive application. In this scenario, if the owner does not click on the controls in first window 103 for a certain period of time, the phone may continue to run sensitive applications, such as WeChat, in the foreground. This means that non-owners can use sensitive applications with the owner's permission.

[0180] The following describes the usage scenarios of the embodiments of this application:

[0181] The embodiments of the present application are applicable to a scenario where the owner gives the first device to a non-owner for use, and the owner wishes to restrict the non-owner's access to the first device, that is, restrict the non-owner from using sensitive applications therein.

[0182] Combine Figure 2 The device control method provided in the embodiment of the present application can be applied to a device management system including a first device 201 and a second device 202. Figure 2 The system architecture provided in the embodiments of the present application.

[0183] Figure 2 In the embodiment, the first device 201 and the second device 202 can be communicatively connected.

[0184] It is understandable that the first device 201 and the second device 202 may be connected via a near-field connection, for example, via Bluetooth; or may be connected via a far-field connection, for example, via a Wireless Local Area Network (WLAN).

[0185] Figure 2 In the embodiment, the first device 201 may be a terminal device. The terminal device may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or the like.

[0186] The second device 202 may be a wearable device, which may be a smart watch or a smart bracelet, etc., and is not limited in this embodiment of the present application.

[0187] Combine Figure 1 , the first device may be a mobile phone, and the second device may be a watch.

[0188] Figure 3A It is a software structure block diagram of the terminal device in an embodiment of the present application.

[0189] Reference Figure 3A The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the terminal device.

[0190] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0191] like Figure 3A As shown, the application layer can include a series of applications. As an example, the application layer can include multiple applications such as sports health, WeChat, King of Glory, QQ, Alipay, Bluetooth, WLAN, etc.

[0192] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0193] like Figure 3A As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a power manager, and the like.

[0194] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0195] Content providers are used to store and retrieve data and make it accessible to applications. Data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0196] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0197] The phone manager is used to provide communication functions for terminal devices, such as call status management (including answering, hanging up, etc.).

[0198] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0199] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, without requiring user interaction. For example, the Notification Manager can be used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating the device, or flashing indicator lights.

[0200] The power manager manages power operations, including sleep, wakeup, restart, and screen brightness adjustment. In practice, when the power manager receives a goToSleep command, the device goes to sleep, effectively locking the screen. When the power manager receives a wakeUp command, the device wakes up, effectively unlocking the screen. When the power manager receives a reboot command, the device restarts.

[0201] The application framework layer may also include collaborative services. Among them, collaborative services are system services. Collaborative services can be used to restrict the use of sensitive applications by non-owners. For example, when a non-owner uses a sensitive application, the collaborative service can send a prompt message (or notification message) to a device paired with the terminal device, such as a watch, through the network to notify the owner of the non-owner's use of the sensitive application through the paired device. In this way, the owner can be informed of the use of the sensitive application in a timely manner, and based on the actual situation, choose whether to execute the operation to prevent the non-owner from using the sensitive application on the paired device.

[0202] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.

[0203] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0204] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0205] The system library can include multiple functional modules, such as the Activity Task Manager Service (ATMS), Surface Manager, Media Libraries, 3D graphics processing library (such as OpenGL ES), and 2D graphics engine (such as SGL).

[0206] The ATMS module is used to manage the startup or switching of applications. It can monitor the startup of applications or the switching of applications from the background to the foreground.

[0207] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0208] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.164, MP3, AAC, AMR, JPG, PNG, etc.

[0209] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0210] A 2D graphics engine is a drawing engine for 2D drawings.

[0211] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0212] Figure 3B This is a software structure diagram of the wearable device according to an embodiment of the present application.

[0213] Reference Figure 3B The software system of the wearable device can adopt a layered architecture, event-driven architecture, micro-kernel architecture, microservice architecture or cloud architecture. The embodiment of the present application takes the software system of the layered architecture as an example to illustrate the software system of the wearable device.

[0214] like Figure 3B As shown, the software system includes several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, such as Figure 3B As shown, the software system can include five layers, which are application layer, application framework layer, system library, hardware abstraction layer and driver layer from top to bottom.

[0215] The application layer may include, for example, health applications, system applications, device management, Bluetooth, WLAN, and other applications.

[0216] In the embodiment of the present application, the application layer may receive the angle data of the rotatable input device reported by the driver layer, so that the application framework layer can obtain the angle data from the application layer.

[0217] The application framework layer provides an API and programming framework for the applications in the application layer. The application framework layer may include some predefined functions.

[0218] For example, the application framework layer may include a front-end framework module (User Interface Kit, UIKIT) and collaborative services.

[0219] Collaborative services are system services. They can be used to restrict the use of sensitive apps by non-owners. For example, upon receiving a notification indicating the use of a sensitive app by a non-owner, the collaborative service can display an interface reminder to the owner. This allows the owner to be informed of the use of sensitive apps and, based on the actual situation, choose whether to perform actions on a wearable device, such as a watch, to prevent non-owners from using sensitive apps.

[0220] Among them, UIKIT is used to build and manage the user interface (UI) graphical interface of the application. In the embodiment of the present application, the UIKIT of the application framework layer can realize the display of the interface required by the collaborative service and transmit the operation information of the user's operation on the controls on the interface to the collaborative service.

[0221] The system library can include multiple functional modules, such as sleep algorithm, motion algorithm, pressure algorithm, basic C library, surface manager, media library, 3D graphics processing library, 2D graphics engine, etc.

[0222] The hardware abstraction layer is used to abstract the hardware. For example, the hardware abstraction layer may include an abstraction layer for the display screen, an abstraction layer for the sensor, and an abstraction layer for other hardware devices (e.g., camera devices, audio devices).

[0223] The driver layer is used to provide drivers for different hardware devices. For example, the driver layer may include display drivers, sensor drivers, and drivers for other hardware devices (e.g., camera devices, audio devices).

[0224] The following takes the first device as a mobile phone and the second device as a watch as an example, and specifically explains the device control method provided in the embodiment of the present application from three parts: establishing a connection between the mobile phone and the watch, setting sensitive applications on the mobile phone, and real-time monitoring of sensitive applications on the mobile phone.

[0225] (1) Establishing a connection between the phone and the watch

[0226] In practice, a mobile phone can establish a connection with a watch in a variety of ways. As an example, when a mobile phone and a watch are first connected, the mobile phone can scan the device identification code displayed on the watch to establish a connection with the watch. The device identification code displayed on the watch is usually a QR code. It is understood that the device identification code can also be in other forms, such as a barcode.

[0227] Combine Figure 4 The phone can also actively search for other devices, first search for the watch, and then establish a connection with the watch. Figure 4 A schematic diagram of the process of establishing a connection between a mobile phone and a watch provided in an embodiment of the present application.

[0228] like Figure 4 As shown in (a), the mobile phone can enter the interface of the sports health application in response to the user clicking the sports health application icon.

[0229] like Figure 4 As shown in (b), the mobile phone can respond to the user clicking on the device interface 401 of the sports health application Control 402, enters the add device interface 403. It can be understood that the mobile phone can also respond to the user clicking the "add device" text operation to enter the add device interface 403.

[0230] like Figure 4 As shown in (c), the mobile phone can search for available devices. Figure 4 From (c), it can be known that the mobile phone can also respond to the user clicking the manual add control 404 to add a watch, and the mobile phone can also respond to the user clicking the scan code to add control 405 to add a watch.

[0231] like Figure 4 As shown in (d) of FIG, the mobile phone scans an available device, specifically the Honor 4 Pro. The mobile phone can further respond to the user clicking the connection control 406 to enter the connection device interface 407. It is understandable that the mobile phone can also respond to the user clicking the text "Honor 4 Pro" to enter the connection device interface 407.

[0232] like Figure 4 As shown in (e) in FIG. 4 , the mobile phone prompts the user to confirm pairing on the mobile phone through the connection device interface 407 .

[0233] like Figure 4 As shown in (f) in FIG. 4 , the watch establishes a connection with the mobile phone in response to the user clicking the “√” control 408 .

[0234] like Figure 4As shown in (g), when the watch and the mobile phone are successfully connected, the watch displays a pairing success interface to prompt the user that the mobile phone and the watch have been successfully paired.

[0235] like Figure 4 As shown in (h), when the watch and the mobile phone are successfully connected, the mobile phone displays a pairing success interface to prompt the user that the mobile phone and the watch have been successfully paired.

[0236] It's understandable that after the phone and watch are first paired and connected, if they are not in the same network range, for example, if the phone is at work but the watch is at home, the phone and watch may lose connection. Similarly, if the phone and watch are no longer in the same network range and then return to the same network range, the phone and watch will automatically reconnect.

[0237] (2) Setting up sensitive applications on your phone

[0238] Here, the mobile phone may respond to the user's setting operation on the target application and determine the application set by the setting operation as a sensitive application.

[0239] Among them, the target application is usually a sports and health application. In some application scenarios, the target application can also be other applications, for example, it can be a "settings" application (an application specifically used to perform system setting functions). It can be understood that the embodiment of the present application does not specifically limit the target application.

[0240] The following uses the target application, a sports and health application, as an example to illustrate the process of setting up sensitive applications on a mobile phone.

[0241] (1) How to add sensitive applications.

[0242] Figure 5A This is a schematic diagram of the process of adding a sensitive application to a mobile phone for the first time, provided in an embodiment of the present application. It should be noted that adding a sensitive application for the first time generally refers to adding a sensitive application when no sensitive application already exists, which may involve the following two scenarios: Scenario 1: The mobile phone has never added a sensitive application; Scenario 2: The mobile phone has previously added sensitive applications but has deleted them.

[0243] like Figure 5AAs shown in (a), the mobile phone can perform owner identification processing in response to the user clicking the sensitive application use reminder switch 502 in the "OFF" state on the sensitive application setting interface 501. Among them, the initial state of the sensitive application use reminder switch 502 is "OFF", that is, the closed state. The sensitive application use reminder switch 502 is used to turn on or off the sensitive application use reminder function (or called the application reminder function). Specifically, when the sensitive application use reminder switch 502 is on, the mobile phone can turn on the sensitive application use reminder function, and when the sensitive application use reminder switch 502 is off, the mobile phone can turn off the sensitive application use reminder function. Among them, the sensitive application setting interface 501 is an interface for setting sensitive applications in the target application (for example, a sports and health application). It can be understood that the embodiment of the present application does not specifically limit how to enter the sensitive application setting interface 501.

[0244] Considering that a phone may be connected to multiple watches, users can set whether to enable sensitive app usage reminders for each watch. As you can understand, the user's sensitive app setting operations for each watch are basically the same, so I will not go into details here.

[0245] like Figure 5A As shown in (b), the mobile phone owner identification is successful, that is, the mobile phone recognizes the owner.

[0246] like Figure 5A As shown in (c), if the phone successfully identifies the owner, it switches the sensitive app usage reminder switch 502 from the original "OFF" state to the "ON" state, that is, from off to on. It can be understood that when the sensitive app usage reminder switch 502 is on, the phone can detect the use of sensitive apps and promptly notify the owner of the sensitive app usage through the watch paired with the phone.

[0247] Afterwards, if Figure 5A As shown in (c) of FIG. 5 , the mobile phone can further respond to the user clicking the add application control 503 on the sensitive application setting interface 501 by entering the add application interface 504. It is understood that if the owner identification fails, that is, if the owner is not identified, the mobile phone maintains the sensitive application usage reminder switch 502 in the original "OFF" state, that is, it is still in the disabled state.

[0248] like Figure 5AAs shown in (d) of FIG, multiple applications are displayed in application interface 504. Initially, the switches of each application displayed in application interface 504 are in the "OFF" state, that is, they are not set as sensitive applications. The mobile phone can further respond to the user clicking the "OFF" switches of the Alipay application, the Honor of Kings application, and the WeChat application on application interface 504 to add Alipay, Honor of Kings, and WeChat as sensitive applications.

[0249] like Figure 5A As shown in (e), the mobile phone switches the status of the application switches of Alipay, Honor of Kings and WeChat in the application interface 504 from the original "OFF" state to the "ON" state, that is, the mobile phone has added Alipay, Honor of Kings and WeChat as sensitive applications.

[0250] Afterwards, if Figure 5A As shown in (e) of FIG. 5 , the mobile phone may further respond to the user clicking the “<” control on the add application interface 504 to enter the sensitive application setting interface 501. It is understandable that the mobile phone may also respond to the user clicking the “add application” text operation to enter the sensitive application setting interface 501.

[0251] like Figure 5A As shown in (f) in the figure, the phone successfully adds Alipay, Honor of Kings, and WeChat, which the user selected, as sensitive apps. In some application scenarios, adding an app as a sensitive app can be considered adding the app to a preset app list. Apps in the preset app list are sensitive apps.

[0252] It is understandable that since non-owners usually do not turn on the sensitive application usage reminder for themselves on other people's phones, in some application scenarios, the phone may not perform owner recognition in response to the user clicking the sensitive application usage reminder switch 502 in the "OFF" state. Figure 5A For example, the mobile phone can directly Figure 5A In the interface (a), jump to Figure 5A The interface (c) in the example is to skip Figure 5A (b) is the owner identification interface.

[0253] (2) How to adjust sensitive applications.

[0254] It is considered that after setting sensitive applications, users may need to adjust sensitive applications, for example, they may continue to add sensitive applications, or they may delete some or all of the sensitive applications that have been set.

[0255] The following further combines Figure 5B and Figure 5C Explain how to adjust sensitive applications that have already been set up.

[0256] in, Figure 5B A schematic diagram of the process of deleting sensitive applications on a mobile phone provided in an embodiment of the present application. Figure 5C A schematic diagram of another process of deleting sensitive applications on a mobile phone provided in an embodiment of the present application.

[0257] like Figure 5B As shown in (a), the mobile phone can perform owner identification in response to the user clicking the management application control 505 on the sensitive application setting interface 501.

[0258] like Figure 5B As shown in (b), the mobile phone owner identification is successful, that is, the mobile phone recognizes the owner.

[0259] like Figure 5B As shown in (c), when the mobile phone successfully identifies the owner, it can enter the management application interface 506.

[0260] Afterwards, if Figure 5B As shown in (c) of FIG, in response to the user clicking the "ON" switch of the Alipay application on the management application interface 506, the mobile phone can switch the Alipay application switch from on to off, that is, restore Alipay to a non-sensitive application. It can be understood that restoring an application originally set as a sensitive application to a non-sensitive application can also be described as deleting the sensitive application or removing the sensitive application from the preset application list.

[0261] It should be noted that if the owner identification fails, that is, if the owner is not identified, the mobile phone will not enter the management application interface 506 , that is, the mobile phone continues to stay in the sensitive application setting interface 501 .

[0262] like Figure 5B As shown in (d), the mobile phone enters the sensitive application setting interface 501 in response to the user clicking the "<" control on the management application interface 506. It is understandable that the mobile phone can also enter the sensitive application setting interface 501 in response to the user clicking the "manage application" text operation.

[0263] like Figure 5B As shown in (e), the mobile phone successfully deleted Alipay from multiple sensitive applications, that is, from the default application list.

[0264] Combine Figure 5B (a) to Figure 5B As can be seen from (e), after entering the application management interface 506, the mobile phone can respond to the user's opening or closing operations on each application and add or delete sensitive applications in batches.

[0265] Further integration Figure 5C ,like Figure 5C As shown in (a), the mobile phone can perform owner identification in response to the user clicking the Alipay application switch in the "ON" state on the sensitive application setting interface 501.

[0266] like Figure 5C As shown in (b), the mobile phone owner identification is successful, that is, the mobile phone recognizes the owner.

[0267] like Figure 5C As shown in (c) of the figure, when the phone recognizes the owner, it switches the Alipay application switch on the sensitive application setting interface 501 from "ON" to "OFF", that is, it restores the Alipay application, which was originally a sensitive application, to a non-sensitive application. It is understandable that when the phone does not recognize the owner, the Alipay application switch on the sensitive application setting interface 501 remains "ON".

[0268] Combine Figure 5C As shown in (c) in FIG, when the mobile phone responds to the user's closing operation of closing an application (e.g., the Alipay application) on the sensitive application setting interface 501, the mobile phone switches the application switch of the application on the sensitive application setting interface 501 to "OFF". In other words, when the status of the application switch of an application on the sensitive application setting interface 501 is "OFF", it means that the application has been deleted from the preset application list.

[0269] It is understandable that in some application scenarios, when the mobile phone responds to the user's closing operation of closing an application (for example, Alipay application) on the sensitive application setting interface 501, the application that has been restored to a non-sensitive application may not be displayed on the sensitive application setting interface 501. Figure 5C Taking (c) in the example, it is not necessary to continue to display the application switch of the Alipay application.

[0270] It should be pointed out that when a mobile phone responds to a user's operation to delete a sensitive application, it first performs owner identification. Only when the owner is identified will the sensitive application be deleted. This can avoid the situation where a non-owner intentionally deletes the sensitive application, resulting in the owner being unable to promptly know that the non-owner is using the sensitive application.

[0271] Combine Figure 5C (a) to Figure 5C As can be seen from (c) in FIG. 5 , the mobile phone can respond to the user's operation of turning on or off sensitive applications directly on the sensitive application setting interface 501 , and add or delete sensitive applications.

[0272] It is understandable that when the user turns on or off a sensitive application directly on the sensitive application setting interface 501, the user may operate multiple sensitive application switches on the sensitive application setting interface 501 at the same time, and the operation interval for multiple sensitive application switches is usually not too long, for example, 20 seconds. Therefore, the mobile phone usually only performs owner identification when responding to the application switch that the user first operates, which can save unnecessary loss of computing resources.

[0273] Considering that non-owners generally do not intentionally add sensitive applications, in some application scenarios, when a user directly turns on or off a sensitive application on the sensitive application setting interface 501, the mobile phone may only perform owner identification in response to the user turning off the first application switch, thereby further saving unnecessary computing resource consumption. The first application switch is generally the first application switch operated by the user within a period of time.

[0274] (3) How to turn off the reminder function for using sensitive applications.

[0275] Considering that users may need to disable the sensitive app usage reminder function after enabling it, for example, when using the phone alone, the user can disable the sensitive app usage reminder function. This not only avoids the need for the user to cooperate with the owner identification process, which helps improve the user experience, but also reduces the consumption of computing resources for owner identification, thereby improving the overall responsiveness of the phone.

[0276] The following combination Figure 5D Explain how to turn off the reminder function for sensitive applications. Figure 5D A schematic diagram of the process of disabling the sensitive application usage reminder function provided in an embodiment of the present application.

[0277] like Figure 5D As shown in (a) in FIG, the mobile phone performs owner identification in response to the user clicking the sensitive application usage reminder switch 502 in the "ON" state on the sensitive application setting interface 501.

[0278] like Figure 5D As shown in (b), the mobile phone owner identification is successful, that is, the mobile phone recognizes the owner.

[0279] like Figure 5DAs shown in (c) of the figure, upon recognizing the owner, the phone switches the sensitive app usage reminder switch 502 from "ON" to "OFF," and the application switches of each sensitive application also switch from "ON" to "OFF." In other words, when the owner turns off the sensitive app usage reminder function, they also delete the sensitive applications that have been set. It is understandable that in some application scenarios, when the phone owner subsequently turns on the sensitive app usage reminder function, the phone can also simultaneously restore the applications on the sensitive application setting interface 501 to sensitive applications.

[0280] It should be pointed out that when the mobile phone responds to the user's operation to turn off the sensitive application usage reminder function, it first performs owner identification. Only when the owner is identified will the sensitive application usage reminder function be turned off. This can avoid the situation where the owner is unable to promptly know that a non-owner is using sensitive applications due to the non-owner deliberately turning off the sensitive application usage reminder function.

[0281] (4) How to view sensitive application usage records.

[0282] In the embodiments of the present application, it is considered that a user may need to comprehensively view the usage of sensitive applications by non-users. For example, a parent may want to better understand their child and promptly identify any potential problems by viewing their child's usage of sensitive applications over a period of time.

[0283] The following further combines Figure 5E Explain how to view the usage of sensitive applications. Figure 5E A schematic diagram of the process of viewing sensitive application usage records provided in an embodiment of the present application.

[0284] like Figure 5E As shown, the mobile phone can respond to the user clicking on the sensitive application setting interface 501. The operation of control 507 expands and displays the record of sensitive application usage. It is understandable that the mobile phone can also expand and display the record of sensitive application usage in response to the user clicking the text "Sensitive application usage record".

[0285] Understandably, the phone can also respond to user clicks The operation of control 507 displays the record of sensitive application usage on another interface. In other words, the embodiment of the present application does not specifically limit how to display the record of sensitive application usage.

[0286] It should be pointed out that Figure 5E The mobile phone displays the sensitive application usage record 507 on the sensitive application setting interface 501, so that the user can clearly view the sensitive application usage record when setting the sensitive application, which helps to improve the user experience.

[0287] Combine 5A to 5D It can be seen that operations such as adding sensitive apps, adjusting sensitive apps, turning on or off the sensitive app usage reminder function, and viewing sensitive app usage records can all be performed on the sensitive app settings interface 501. The same interface supports multiple operations, making it easier for users to set up sensitive apps and view sensitive app usage records, thereby improving the user experience.

[0288] (5) How to perform owner identification.

[0289] In practice, the mobile phone needs to perform owner identification to determine whether the user of the mobile phone is the owner, and then perform the corresponding response action (combined with 5A to 5D ). The following describes the owner identification solution of the embodiment of the present application. Figure 6A This is a flowchart of the owner identification process provided by the embodiment of the present application. It is understandable that the mobile phone usually completes the owner identification by calling the owner identification interface on the mobile phone. When the owner identification interface is called, it can execute Figure 6A The owner identification scheme shown.

[0290] S601: The mobile phone determines whether a face is detected. If no face is detected, S602 is executed. If a face is detected, S605 is executed.

[0291] Here, the phone can analyze whether the detected image is a face image, for example, by analyzing the image's pixel values. If it is a face image, the phone can combine it with a pre-recorded face template of the owner to identify the owner's face and obtain a face recognition score. The face recognition score describes the degree of similarity between the face under test and the face template. The face under test is the face of the user to be tested (or referred to as the user under test).

[0292] In step S602, if no face is detected, the mobile phone determines whether breathing sounds are detected. If breathing sounds are detected, step S603 is executed. If no breathing sounds are detected, it indicates that the detection result is unreliable, and step S604 is executed.

[0293] In an embodiment of the present application, if no face is detected, the mobile phone can continue to collect the user's breathing audio data and extract the breathing sound features in the breathing audio data. This can then be used to determine whether the user is the owner of the device by comparing the breathing sound features with the breathing sound feature template in the device owner template. The breathing audio data is the user's breathing audio data. The breathing sound features are the breathing sound features in the breathing audio data.

[0294] Since the mobile phone may recognize that the collected breathing audio data to be tested does not contain breathing sounds, if the mobile phone recognizes that the breathing audio data to be tested does not contain breathing sounds, it is considered that no breathing sounds are detected. Conversely, if the mobile phone recognizes that the breathing audio data to be tested contains breathing sounds, it is considered that breathing sounds are detected.

[0295] S603: Identify the device owner based on the breath sound characteristics to be measured, and output the device owner identification result.

[0296] In the embodiment of the present application, in order to distinguish the description, the breathing sound characteristics of the user to be tested can be recorded as the breathing sound characteristics to be tested.

[0297] Here, when breathing sounds are detected in S602 , it means that the mobile phone recognizes that the breathing audio data to be measured contains breathing sounds, and the mobile phone can extract and output the features of the breathing sounds to be measured.

[0298] Afterwards, the mobile phone can use the breathing sound characteristics corresponding to the user to be tested to identify whether the user to be tested is the owner of the phone.

[0299] Figure 6B The figure shows the process of identifying the owner of the device based on breathing sounds. Figure 6B The mobile phone can compare the breathing sound characteristics of the user to be tested corresponding to the breathing sound characteristics of the user to be tested with the pre-stored breathing sound characteristic template of the owner of the phone to determine whether the user to be tested is the owner of the phone.

[0300] In some optional implementations of the embodiments of the present application, the mobile phone can determine whether the user to be tested is the owner of the device by: when the owner template includes a breathing sound feature template, determining the similarity between the breathing sound feature to be tested and the breathing sound feature template; if the obtained similarity is greater than a pre-set second determination threshold, determining that the user to be tested is the owner of the device. Conversely, if the obtained similarity is less than or equal to the second determination threshold, determining that the user to be tested is not the owner of the device.

[0301] The second determination threshold is usually a preset similarity value, for example, 0.6, and the value range of the second determination threshold is usually 0-1.

[0302] In practice, the similarity between the respiratory sound feature to be measured and the respiratory sound feature template is calculated by generally calculating the cosine similarity between the vector corresponding to the respiratory sound feature to be measured and the vector corresponding to the respiratory sound feature template.

[0303] In some optional implementations of the embodiments of the present application, the mobile phone may also determine whether the user to be tested is the owner of the phone by: when the owner template includes multiple breathing sound feature templates, the similarity between each breathing sound feature template and the breathing sound feature to be tested is determined. When the average of the multiple similarities is greater than a preset fourth determination threshold, the user to be tested is determined to be the owner of the phone. Conversely, when the average of the multiple similarities is less than or equal to the fourth determination threshold, the user to be tested is determined to be a non-owner of the phone.

[0304] The fourth determination threshold is usually a preset similarity value, for example, 0.3, and the value range of the fourth determination threshold is 0-1.

[0305] To give a further example, if there are three breathing sound feature templates in the owner template, namely breathing sound feature template 1, breathing sound feature template 2 and breathing sound feature template 3, if the similarity between the breathing sound feature to be tested and breathing sound feature template 1 is S1, the similarity between the breathing sound feature to be tested and breathing sound feature template 2 is S2, and the similarity between the breathing sound feature to be tested and breathing sound feature template 3 is S3, then when the average of S1, S2 and S3 is greater than the fourth judgment threshold, the user to be tested is considered to be the owner of the device, otherwise, the user to be tested is considered to be a non-owner of the device.

[0306] It should be pointed out that when there are multiple breathing sound feature templates in the owner template, the accuracy of owner identification can be improved by judging whether the user to be tested is the owner by taking the average of the similarities between the breathing sound feature to be tested and each breathing sound feature template.

[0307] Optionally, if the owner template includes multiple breathing sound feature templates, the mobile phone may also determine the similarity between each breathing sound feature template and the breathing sound feature to be tested. When the maximum of the multiple similarities obtained is greater than a second determination threshold, the user to be tested is determined to be the owner. Conversely, when the maximum of the multiple similarities obtained is less than or equal to the second determination threshold, the user to be tested is determined to be a non-owner.

[0308] In some optional implementations of the embodiments of the present application, when multiple owner templates are recorded in the mobile phone, the mobile phone may also determine whether the user to be tested is the owner in the following manner: for each owner template, when the owner template includes a breathing sound feature template, determining the similarity between the breathing sound feature to be tested and the breathing sound feature template; when the obtained similarity is greater than a second determination threshold, determining that the user to be tested is the owner, and specifically the owner indicated by the owner template; when the owner template includes multiple breathing sound feature templates, respectively determining the similarity between each breathing sound feature template in the owner template and the breathing sound feature to be tested; when the average of the multiple similarities obtained is greater than a fourth determination threshold, determining that the user to be tested is the owner, and specifically the owner indicated by the owner template;

[0309] Optionally, when the user to be tested is not the owner indicated by any of the owner templates, the mobile phone may determine that the user to be tested is not the owner. In other words, the mobile phone considers that the owner has not been identified.

[0310] In an embodiment of the present application, when the user to be tested is identified as the owner, or when the mobile phone recognizes the owner, the mobile phone can continue to perform operations related to the owner's permissions, such as running sensitive applications in the foreground, turning on or off the reminder function for using sensitive applications, deleting sensitive applications, adding sensitive applications, etc.

[0311] S604, prompting that the test result is unknown.

[0312] In the embodiment of the present application, if the mobile phone detects neither a face nor breathing sounds during the owner identification process, the detection result can be considered unreliable. In this case, the mobile phone can output a prompt message such as "Detection result unknown", "Retry once", or "Try again".

[0313] Afterwards, the user to be tested can combine the prompts of the mobile phone to further cooperate with the mobile phone to identify the owner.

[0314] S605: The mobile phone inputs the face recognition score into a threshold decision device to determine whether the user to be tested is the owner of the phone.

[0315] Here, if the facial recognition score is greater than a preset upper threshold for facial recognition, such as 0.65, the user under test is considered the owner of the device. If the facial recognition score is less than a preset lower threshold for facial recognition, such as 0.05, the user under test is considered a non-owner. If the facial recognition score is between the lower and upper thresholds, step S606 can be continued.

[0316] The face recognition upper threshold may be a FAR threshold for face recognition. The face recognition lower threshold may be a FRR threshold for face recognition. In practice, a face recognition score between the face recognition lower threshold and the face recognition upper threshold generally indicates a poor face detection result.

[0317] In step S606, if the face detection result is not good, the mobile phone determines whether breathing sound is detected. If no breathing sound is detected, step S607 is executed. Conversely, if breathing sound is detected, step S608 is executed.

[0318] In an embodiment of the present application, when the face detection result is poor, the mobile phone can continue to collect the breathing audio data to be tested of the user to be tested, and process the breathing audio data to be tested to extract the breathing sound features to be tested in the breathing audio data to be tested.

[0319] In practice, if the mobile phone recognizes that the breathing audio data to be tested does not contain breathing sounds, it is considered that no breathing sounds are detected. Conversely, if the mobile phone recognizes that the breathing audio data to be tested contains breathing sounds, it is considered that breathing sounds are detected.

[0320] S607: When no breathing sound is detected, the mobile phone directly outputs the face detection result.

[0321] Here, the mobile phone combines the pre-recorded face template of the owner to detect and recognize the face, and can combine the face recognition score to make a preliminary judgment on whether it is the owner. If no breathing sound is detected, the mobile phone can output this preliminary judgment. For example, if the face recognition score is greater than a pre-set face recognition score threshold, the person is judged to be the owner. Conversely, if the face recognition score is less than or equal to the face recognition score threshold, the person is judged to be a non-owner. The above-mentioned face recognition score threshold can be a pre-set threshold, for example, it can be 0.5.

[0322] Optionally, in some application scenarios where the accuracy of owner recognition is very high, if the face detection result is poor and no breathing sound is detected, the mobile phone can also output a message prompting re-detection, for example, it can output "Please try again".

[0323] S608: When breathing sounds are detected, the mobile phone combines the multimodal data of the user to be tested, calls a multimodal fusion owner recognition model, and performs owner recognition.

[0324] Among them, the multimodal data may include face recognition scores, respiratory sound features to be tested, touch screen data to be tested, and IMU data to be tested.

[0325] In the embodiment of the present application, only when the face recognition effect is poor, for example, in the presence of strong light, backlight, dim light, etc., which leads to poor face recognition, will further collect the user's breathing audio data, touch screen data, and IMU data to be tested, so as to combine the multimodal data to jointly perform the machine owner identification to improve the accuracy of machine owner identification. Since the accuracy of face recognition is already high enough most of the time, that is, the use of multimodal data for machine owner identification is relatively rare, it is possible to save the computing power of the mobile phone and reduce the power consumption of the mobile phone while ensuring the accuracy of machine owner identification, thereby improving the practicality of the mobile phone and helping to further enhance the user experience.

[0326] In order to distinguish the description, the touch screen data collected during the owner identification process can be recorded as the touch screen data to be tested, and the collected IMU data can be recorded as the IMU data to be tested.

[0327] It is understood that in some optional implementations, in S608, if breathing sounds are detected, the mobile phone may directly use the breathing sound characteristics of the user to identify the owner. In this case, S608 may be replaced by the following steps: if breathing sounds are detected, the mobile phone identifies the owner based on the breathing sound characteristics and outputs the owner identification result. This operation is essentially the same as the aforementioned S603 and will not be further described here.

[0328] Figure 6C A schematic diagram of the process of machine owner identification using a multimodal fusion machine owner identification model is shown.

[0329] S6081: Calculate the cross-modal features to be measured.

[0330] The cross-modal features to be measured are cross-modal features corresponding to the user to be measured.

[0331] Here, the phone can use a cross-modal encoder to encode the touchscreen data, IMU data, and breathing sound features to generate cross-modal features. The cross-modal encoder encodes the touchscreen data, IMU data, and breathing sound features to generate cross-modal features. Cross-modal features are used to describe the comprehensive characteristics of the device owner across multiple modalities.

[0332] S6082: Calculate the cross-modal similarity between the cross-modal feature to be tested and the cross-modal feature template.

[0333] The cross-modal similarity is the similarity between the cross-modal feature to be tested and the cross-modal feature template. In the embodiment of the present application, the cross-modal similarity can also be referred to as a cross-modal recognition score.

[0334] S6083: Calculate the breathing sound similarity between the breathing sound feature to be measured and the breathing sound feature template of the user.

[0335] The breathing sound similarity is the similarity between the breathing sound feature to be measured and the breathing sound feature template of the owner. In the embodiment of the present application, the breathing sound similarity can also be called a breathing recognition score.

[0336] S6084: Input the cross-modal similarity, breathing sound similarity, and face recognition score into the fusion decision model to obtain a multimodal recognition score.

[0337] The fusion decision model takes as input the cross-modal similarity (also known as the cross-modal recognition score), the breathing sound similarity (also known as the breathing recognition score), and the face recognition score, and outputs a multimodal recognition score. The multimodal recognition score describes the comprehensive recognition results of the user under test in multiple modalities. The fusion decision model is used to fuse the cross-modal recognition score, the breathing recognition score, and the face recognition score to produce a single multimodal recognition score.

[0338] S6085: Determine whether the user to be tested is the device owner based on the multimodal recognition score.

[0339] In an embodiment of the present application, if the multimodal recognition score is greater than or equal to a preset fusion threshold, the user is identified as the owner; if the multimodal recognition score is less than the fusion threshold, the user is identified as a non-owner.

[0340] The above fusion threshold is usually a preset threshold, for example, it can be 0.6.

[0341] Combine Figure 6C It can be seen that the multimodal fusion machine owner recognition model uses a cross-modal encoder to fuse IMU data, touch screen data, and breathing sound features at the feature level. The decision fusion model then fuses the cross-modal recognition score, breathing recognition score, and face recognition score at the decision level to obtain the multimodal recognition score. Among them, stronger modalities, such as the face modality, are not involved in feature-level fusion. This reduces coupling between data and effectively utilizes the complementary advantages of multiple modalities, improving the model's overall recognition success rate.

[0342] Considering that a mobile phone may have an owner template with one owner or multiple owners, that is, a mobile phone may have one owner template or multiple owner templates.

[0343] In some optional implementations of the present application, when multiple master templates are pre-recorded in the mobile phone, combined with the above Figure 6C During the process of executing S6081-S6084, the mobile phone may first select a host template and calculate the multimodal recognition score using the selected host template.

[0344] In some application scenarios, the mobile phone can select the owner template with the highest corresponding face recognition score. For example, if there are three owner templates, namely owner template 1, owner template 2, and owner template 3, and the mobile phone identifies the user to be tested as the owner corresponding to owner template 1 during S601, owner template 1 can be directly selected.

[0345] In this application scenario, the mobile phone calculates a multimodal recognition score for owner template 1 during steps S6081-S6084. The mobile phone can then use this multimodal recognition score to determine if the user is the owner. If so, the user is the owner indicated by owner template 1. This application scenario requires only calculating a multimodal recognition score for a single owner template, resulting in a relatively low computational load.

[0346] In other optional implementations of the embodiments of the present application, if multiple owner profiles are pre-registered in the mobile phone, the mobile phone can calculate a multimodal recognition score for each owner profile. The mobile phone can then combine the multiple multimodal recognition scores to determine whether the user being tested is the owner and which owner it is.

[0347] To give a further example, if there are three owner templates, namely owner template 1, owner template 2 and owner template 3, the mobile phone can use owner template 1 to perform S6081-S6084 operations to obtain a multimodal recognition score, such as 99 points, use owner template 2 to perform S6081-S6084 operations to obtain another multimodal recognition score, such as 78 points, and use owner template 3 to perform S6081-S6084 operations to obtain another multimodal recognition score, such as 60 points. At this time, it can be determined that the user to be tested is the owner, and specifically the owner indicated by owner template 1.

[0348] In this implementation, step S608, where the multimodal fusion model is used to further identify the device owner, is performed only when face detection is ineffective. If face detection in step S601 is ineffective, selecting the device owner template based on the face detection result may be inaccurate. Therefore, in this implementation, if face detection accuracy in S601 is not high, a multimodal recognition score is calculated for each device owner template separately, ensuring the accuracy of device owner identification.

[0349] Consider that a host template may contain multiple feature templates, for example, multiple face templates, multiple breathing sound feature templates, or multiple cross-modal feature templates.

[0350] In some optional implementations of the embodiments of the present application, when there are multiple feature templates in the host template, when the mobile phone executes S6081-S6084, the mean of the multiple feature templates in the host template can be calculated for similarity with the features to be measured of the same modality (for example, the breathing sound features to be measured) to obtain the recognition score of the corresponding modality.

[0351] For example, if the phone owner's template contains three breathing sound feature templates, the phone can calculate the similarity between the average of the three breathing sound feature templates and the breathing sound feature to be tested to obtain the recognition score corresponding to the breathing mode, or called the breathing recognition score.

[0352] Optionally, the phone can also calculate the similarity between each feature template in the owner template and the feature to be tested of the same modality, thereby obtaining multiple similarities. The recognition score for the corresponding modality can then be calculated based on these multiple similarities. For example, the maximum value among the multiple similarities can be used as the recognition score for the corresponding modality.

[0353] To give a further example, if the owner's template contains three breathing sound feature templates, the mobile phone can calculate the similarity between each breathing sound feature template and the breathing sound feature to be tested to obtain three similarities. Afterwards, the maximum value of the three similarities can be used as the recognition score corresponding to the breathing mode, or called the breathing recognition score.

[0354] In some optional implementations of the present application, when multiple master templates are pre-recorded in the mobile phone, combined with the above Figure 6C During S6085, since each owner template may correspond to a multimodal recognition score, the phone can determine whether the user under test is the owner by: For each multimodal recognition score corresponding to each owner template, if the multimodal recognition score is greater than the fusion threshold, the phone can determine that the user under test is the owner. In this case, the user under test can be identified as the owner indicated by the owner template corresponding to the multimodal recognition score.

[0355] In some application scenarios, if multiple multimodal recognition scores are simultaneously greater than the fusion threshold, the phone can determine that the user being tested is the owner. In this case, the user being tested can be identified as the owner indicated by the owner template corresponding to the largest of the multiple multimodal recognition scores.

[0356] (3) Real-time monitoring of sensitive applications on mobile phones

[0357] Here, the mobile phone can be a collaborative service in the mobile phone (combined with Figure 3A) can detect in real time whether sensitive apps are being used. When a sensitive app is in use, the phone can send a notification to the paired watch, which promptly notifies the owner of any attempt by a non-owner to use a sensitive app on their phone. This way, upon learning that a non-owner is using a sensitive app on their phone, the owner can choose to prevent or not prevent the non-owner from using the app. In other words, the owner can use their watch to conveniently and flexibly restrict non-owners from using sensitive apps on their phone.

[0358] It should be noted that when a mobile phone detects that a sensitive application is being used, it usually means that the mobile phone detects that a sensitive application is started or detects that a sensitive application is switched from the background to the foreground.

[0359] The following combination Figure 7 This paper describes how mobile phones and watches can work together to restrict the use of sensitive applications by non-owners. Figure 7 This is a timing diagram of the collaborative operation of a mobile phone and a watch provided in an embodiment of the present application.

[0360] S701: The ATMS module in the mobile phone responds to the user's first operation and determines that an application startup or switch is detected.

[0361] The first operation is usually used to trigger the application to run in the foreground.

[0362] In practice, the first operation may be an operation of starting an application (or referred to as a start operation), for example, it may be an operation of clicking an application icon.

[0363] The first operation may also be an operation to confirm switching the application from the background to the foreground (or called a switching operation), for example, it may be an operation of clicking a confirmation jump control, where the confirmation jump control is used to confirm whether to jump from the interface of one application to the interface of another application.

[0364] It is understandable that the embodiment of the present application does not specifically limit the implementation form of the first operation.

[0365] Combine Figure 3A The ATMS module is a module in the Android system that manages the startup or switching of applications. The ATMS module can monitor the event of any application startup or application switching from the background to the foreground on the mobile phone. For ease of description, the embodiments of the present application may refer to the event of application startup or application switching from the background to the foreground as an application running event.

[0366] S702: The ATMS module in the mobile phone sends the monitored application running event to the first collaborative service in the mobile phone.

[0367] The first collaborative service is a collaborative service running in the mobile phone. Figure 3A Collaborative services can be used to restrict the use of sensitive apps by non-owners. For example, if a non-owner uses a sensitive app on their phone, the collaborative service can send a notification to the watch paired with the phone, alerting the owner through the watch that the non-owner is accessing the sensitive app. This way, the owner is immediately notified of the non-owner's use of the sensitive app and can choose to block or not block the non-owner from using the sensitive app based on the specific situation.

[0368] It should be pointed out that if the application or service on the mobile phone wants to obtain the above-mentioned application running events monitored by the ATMS module, it usually needs to register with the ATMS module in advance. In other words, the first collaborative service needs to complete the registration with the ATMS module first. After that, the ATMS module will send the above-mentioned application running events monitored in real time to the first collaborative service. In some application scenarios, the application or service can complete the registration with the ATMS module by calling the corresponding function plug-in. It should be pointed out that the application or service can use the existing method to register with the ATMS module, which will not be elaborated here.

[0369] In practice, combined Figures 5A-5E When the sensitive app usage reminder function is turned on (that is, the sensitive app usage reminder switch 502 is on), the first collaborative service on the mobile phone starts running. When the sensitive app usage reminder function is turned off (that is, the sensitive app usage reminder switch 502 is off), the first collaborative service on the mobile phone ends running. In other words, when the first collaborative service is no longer needed, the first collaborative service can be shut down, thereby saving unnecessary resource consumption on the mobile phone and improving the overall data processing efficiency of the mobile phone.

[0370] S703: The first collaborative service on the mobile phone determines whether the application involved in the application running event sent by the ATMS module is a sensitive application.

[0371] In the embodiment of the present application, in order to distinguish the description, the application involved in the application running event may be referred to as the current application.

[0372] In practice, when the mobile phone performs the operation of setting sensitive applications in the above part (2), the sensitive applications can be obtained. For example, a sensitive application list (or preset application list) including one or more sensitive applications can be obtained.

[0373] When the first collaborative service receives an application execution event sent by the ATMS module, it can extract the application involved in the event (i.e., the current application) from the application execution event. The first collaborative service can then compare the current application with each application in the sensitive application list (or preset application list) to determine whether the current application is a sensitive application.

[0374] Here, if the current application is a sensitive application, the first collaborative service can continue to execute S704. It is understandable that the first collaborative service will not continue to execute S704 if it determines that the current application is not a sensitive application. Figure 7 S704 and operations after S704 in .

[0375] S704: The first collaborative service in the mobile phone performs owner identification.

[0376] In practice, the first collaborative service usually performs owner identification by calling the owner identification interface in the mobile phone. For the specific owner identification process, please refer to Figure 6A And the corresponding embodiment part will not be described here.

[0377] Here, if the first collaborative service fails to identify the owner of the device, it may continue to execute S705. It is understandable that if the first collaborative service identifies the owner of the device, it will not continue to execute S705. Figure 7 S705 and operations after S705.

[0378] S705 , when a sensitive application is used and the owner is not identified, the first collaborative service in the mobile phone and the second collaborative service in the watch perform a security negotiation to obtain a first session key.

[0379] The second collaborative service is a collaborative service running on the watch. The first and second collaborative services are typically the same service deployed on different devices. In practice, the first collaborative service on the phone is often used in conjunction with the second collaborative service on the watch to restrict non-users from using sensitive applications.

[0380] The above-mentioned security negotiation is used to negotiate a temporary session key between the phone and the watch. As an example, the phone and the watch can negotiate the temporary session key based on the Simple Password Exponential Key Exchange (SPEKE) protocol. It should be noted that the phone and the watch can use existing temporary session key negotiation technology to negotiate a temporary session key, thereby achieving secure negotiation, which will not be elaborated here.

[0381] In the embodiment of the present application, in order to distinguish the description, the temporary session key obtained by negotiation in S705 may be referred to as the first session key.

[0382] In practice, the information transmission channel between the mobile phone and the watch may include a first channel and a second channel. The first channel is the information transmission channel between the application in the mobile phone (specifically, a sports and health application) and the watch. The second channel is the information transmission channel between the system service in the mobile phone and the watch.

[0383] In practice, the mobile phone usually establishes a connection with the mobile phone through an application in the mobile phone (specifically, a sports health application) (see Figure 4 ), that is, the information transmission channel between the phone and the watch is usually the information transmission channel between the phone's applications and the watch, which is also the first channel mentioned above. Generally, application information can be transmitted between the phone and the watch through the first channel. As an example, the phone can transmit weather information to the watch through the first channel. The watch can also transmit exercise step count information to the phone through the first channel.

[0384] In practice, the first channel and the second channel are usually two channels established when the mobile phone and the watch are connected.

[0385] Optionally, both the first channel and the second channel can communicate based on the Bluetooth transmission protocol. It is understandable that the first channel and the second channel can also communicate based on other transmission protocols, such as WLAN, NFC, etc. The embodiment of the present application does not specifically limit the transmission protocol used by the first channel and the second channel.

[0386] It should be pointed out that since the application may not always be in a live state, that is, it may be closed sometimes. When the application is closed, it may be difficult to ensure timely transmission of information between the mobile phone and the watch. Also, since the system service is usually always in a live state, that is, the second channel can always be open. Therefore, in the embodiment of the present application, the above-mentioned second channel is used to transmit information between the first collaborative service and the second collaborative service, which can ensure that when the mobile phone detects that a sensitive application is being used, the notification information is promptly prompted to the owner through the watch. In addition, since the second channel is an encrypted transmission channel, the security of information transmission between the mobile phone and the watch can be guaranteed.

[0387] S706: The first collaborative service in the mobile phone sends notification information to the second collaborative service in the watch based on the first session key.

[0388] Among them, the above notification information is usually used to indicate that the mobile phone is accessing a sensitive application. It is understandable that the notification information includes the application identifier of the sensitive application. The application identifier can identify the application. The application identifier is usually the application name. Of course, the application identifier can also be in other forms, such as an application icon. As an example, the notification information can be "Alipay" or "Your mobile phone is accessing Alipay." It is understandable that the embodiments of the present application do not specifically limit the implementation form of the notification information.

[0389] It is understood that the information transmitted between the first collaborative service and the second collaborative service is encrypted using the first session key obtained through the secure negotiation in S705. After receiving the notification information, the second collaborative service can decrypt it using the first session key obtained through the secure negotiation in S705 to obtain the information content of the notification information.

[0390] S707: The second collaborative service in the watch sends the notification information to the watch application reminder module in the watch.

[0391] Combine Figure 3B The above-mentioned watch application reminder module is the UIKIT in the watch.

[0392] S708: The watch application reminder module in the watch prompts the user with the notification information.

[0393] Here, the watch, specifically the watch application reminder module in the watch, can display notification information to the user. It should be noted that the user of the watch side that is paired with the mobile phone is usually considered the owner.

[0394] In practice, a watch typically displays notification information to the user, i.e., the owner, by displaying a first interface. The first interface is an interactive interface that includes controls for the owner to operate. For example, it may include a first control and a second control. When the first control is clicked, it instructs the owner to prevent non-owners from using the current sensitive app. When the second control is clicked, it instructs the owner not to prevent non-owners from using the current sensitive app, meaning that the owner consents to non-owners using the current sensitive app.

[0395] Figure 8A Schematic diagram of the first interface displayed on a watch provided in an embodiment of the present application. As shown in 8A, the first window 801 may include notification information 802, a one-touch lock control 803, and a do not remind control 804. When the one-touch lock control 803 is clicked, it indicates that the owner has chosen to prevent non-owners from using the sensitive application currently being accessed. When the do not remind control 804 is clicked, it indicates that the owner does not prevent non-owners from using the sensitive application currently being accessed, that is, the owner agrees that non-owners can use the sensitive application currently being accessed.

[0396] S709: The watch application reminder module in the watch detects the user's locking operation.

[0397] In practice, the watch can detect the user's locking operation through a sensor, such as a touch sensor. Figure 8A The locking operation may be an operation of clicking a key to lock the control 803 .

[0398] S710: The watch application reminder module in the watch calls the lock phone interface of the second collaborative service in response to the user's lock operation.

[0399] The lock phone interface is an interface for locking a phone. When the lock phone interface is called, the second collaborative service can send a lock instruction to the phone, specifically the first collaborative service in the phone. The lock instruction is used to instruct the phone to lock the screen.

[0400] S711: The second collaborative service in the watch performs security negotiation with the first collaborative service in the mobile phone to obtain a second session key.

[0401] In the embodiment of the present application, in order to distinguish the description, the temporary session key obtained by negotiation in S711 is referred to as the second session key. It can be understood that the security negotiation process in S711 is basically the same as the security negotiation process in S705, and will not be repeated here.

[0402] S712: The second collaborative service in the watch sends a lock instruction to the first collaborative service in the mobile phone based on the second session key.

[0403] Among them, the lock command is used to instruct the mobile phone to lock the screen.

[0404] It can be understood that the locking instruction sent by the second collaborative service to the first collaborative service is usually encrypted and transmitted using the second session key.

[0405] S713: The first coordinated service in the mobile phone sends a screen lock instruction to the power manager in the mobile phone.

[0406] The lock screen instruction is used to instruct the power manager to lock the screen. In practice, the lock screen instruction is usually "goToSleep". It can be understood that the embodiment of the present application does not specifically limit the implementation form of the lock screen instruction.

[0407] Here, after receiving the locking instruction, the first collaborative service usually uses the second session key to decrypt the locking instruction, thereby obtaining the content of the locking instruction.

[0408] S714: The first coordinated service in the mobile phone returns a locking result to the second coordinated service in the watch.

[0409] The lock result is used to indicate whether the screen is locked. As an example, the lock result can be "the phone is locked." It is understandable that the embodiment of the present application does not specifically limit the implementation form of the lock result.

[0410] Here, the locking result sent by the first collaborative service to the second collaborative service is usually encrypted and transmitted based on the second session key.

[0411] S715: The second collaborative service in the watch sends the locking result to the watch application reminder module in the watch.

[0412] Here, after receiving the locking result, the second collaborative service usually uses the second session key to decrypt the locking result, thereby obtaining the content of the locking result.

[0413] S716: The watch application reminder module in the watch notifies the user of the locking result.

[0414] The following further combines Figure 8B , describes the interactive scenario of the device control method of the embodiment of the present application. Among them, Figure 8B A schematic diagram of an interactive scenario of a device control method provided in an embodiment of the present application.

[0415] like Figure 8B As shown in (a), the mobile phone can identify the owner in response to the user clicking the application icon 805 of the Alipay application.

[0416] like Figure 8B As shown in (b) in FIG, when the mobile phone fails to identify the owner, a prompt message 806 “Owner identification failed” is displayed on the mobile phone.

[0417] like Figure 8B As shown in (c) of FIG, if the phone fails to recognize the owner, the watch may display a first window 801. The first window 801 may include a notification message 802, a one-touch lock control 803, and a do not remind control 804. Subsequently, the watch may control the phone to lock the screen in response to the user clicking the one-touch lock control 803 on the first window 801.

[0418] like Figure 8B As shown in (d), the phone enters the lock screen state.

[0419] like Figure 8B As shown in (e), when the phone is locked, the watch displays a prompt message 807 of "the phone is locked".

[0420] Combine Figure 8BAs you can see, the phone and watch work together. Specifically, the phone can detect in real time whether sensitive apps are being used. When a sensitive app is used, the phone can send a notification to the paired watch, thus promptly notifying the owner through the watch that a non-owner is attempting to use a sensitive app on their phone. This way, upon learning that a non-owner is using a sensitive app on their phone, the owner can choose to stop or not stop the non-owner from using the sensitive app. In other words, the owner can use the watch to conveniently and flexibly restrict non-owners from using sensitive apps on their phone.

[0421] Corresponding to the device control method of the above embodiment, Figure 9A A structural diagram of the device control system provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0422] Reference Figure 9A , the equipment control system includes:

[0423] The first device 201 performs owner identification processing in response to a first operation of a user triggering a foreground run of a first application;

[0424] The second device 202 displays a first interface when the first device recognizes that the user is not the device owner. The first interface includes a first window. The first window includes a first prompt message and a first control. The first prompt message reminds the user that the first application is being used by a non-device owner.

[0425] The second device 202 controls the first device to lock the screen when the second device receives an operation on the first control by the user.

[0426] Optionally, the first device may be a mobile phone, and the second device may be a wearable device. Optionally, the wearable device may be a watch.

[0427] As an embodiment of the present application, the device control system can be implemented as follows: Figure 1-8B The illustrated embodiments and other related method embodiments.

[0428] The process of each device in the device control system provided in the embodiment of the present application realizing its own function can be specifically referred to the aforementioned Figure 1-8B The description of the illustrated embodiment and other related method embodiments will not be repeated here.

[0429] Corresponding to the device control method of the above embodiment, Figure 9B A structural schematic diagram of the first device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0430] Reference Figure 9B , the first device includes:

[0431] The operation response unit 2011 performs a device owner identification process in response to a first operation triggered by a user to run the first application in the foreground; the notification sending unit 2012 sends a notification message to the second device when it is identified that the user is not the device owner, the notification message indicating that the first application is being used by a non-device owner;

[0432] The lock execution unit 2013 locks the electronic screen in response to receiving the lock instruction returned by the second device in response to the notification information.

[0433] Optionally, the first device may be a mobile phone, and the second device may be a wearable device. Optionally, the wearable device may be a watch.

[0434] As an embodiment of the present application, the first device can implement the following Figure 1-8B The illustrated embodiments and other related method embodiments.

[0435] The process of each module in the first device provided in the embodiment of the present application realizing its own function can be specifically referred to the aforementioned Figure 1-8B The description of the illustrated embodiment and other related method embodiments will not be repeated here.

[0436] Corresponding to the device control method of the above embodiment, Figure 9C A structural schematic diagram of the second device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0437] Reference Figure 9C , the second device includes:

[0438] The notification receiving unit 2021 receives notification information from the first device, where the notification information indicates that the first application on the first device is being used by a non-device owner;

[0439] The interface display unit 2022 displays a first interface in response to the notification information, where the first interface includes a first window, the first window includes a first prompt message and a first control, and the first prompt message reminds that the first application is being used by a non-user;

[0440] The instruction sending unit 2023 sends a lock instruction to the first device in response to the user's operation on the first control, where the lock instruction instructs the first device to lock the screen.

[0441] Optionally, the first device may be a mobile phone, and the second device may be a wearable device. Optionally, the wearable device may be a watch.

[0442] As an embodiment of the present application, the second device can implement the following Figure 1-8B The illustrated embodiments and other related method embodiments.

[0443] The process of each module in the second device provided in the embodiment of the present application realizing its own function can be specifically referred to the aforementioned Figure 1-8B The description of the illustrated embodiment and other related method embodiments will not be repeated here.

[0444] It should be noted that the owner identification, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0445] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0446] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0447] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0448] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0449] In addition, in the description of this application and the appended claims, the terms "first," "second," "third," etc. are used only to distinguish and describe, and should not be understood as indicating or implying relative importance. It should also be understood that although the terms "first," "second," etc. are used in the text to describe various elements in some embodiments of the present application, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0450] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0451] Figure 10A It is a structural diagram of a terminal device provided in one embodiment of the present application.

[0452] The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a camera 193, a touch screen 194, and a SIM card interface 195, etc.

[0453] The audio module 170 may include a speaker and a microphone, etc.

[0454] Among them, the sensor module 180 may include a pressure sensor, an IMU, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, and an ambient light sensor (of course, the terminal device may also include other sensors, such as a temperature sensor, a bone conduction sensor, etc., which are not shown in the figure).

[0455] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. Among them, the controller can be the nerve center and command center of the terminal device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0456] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0457] The processor 110 can execute the device control method provided in the embodiments of the present application. The processor 110 can include different components. For example, when the processor 110 integrates a CPU and a GPU, the CPU and the GPU can cooperate to execute the device control method provided in the embodiments of the present application. For example, part of the algorithm in the device control method is executed by the CPU, and another part of the algorithm is executed by the GPU to achieve faster processing efficiency.

[0458] Figure 10B FIG2 is a schematic diagram of the structure of a wearable device provided in an embodiment of the present application. For example, the wearable device may be a smart watch or a smart bracelet.

[0459] refer to Figure 10B The terminal device may include a processor 210, an internal memory 221, a power supply module 230, an antenna 3, an antenna 4, a mobile communication module 250, a wireless communication module 260, an audio module 270, a sensor module 280, a button 290, a motor 293, a touch screen 294, and a SIM card interface 295, etc.

[0460] The audio module 270 may include a speaker and a microphone, etc.

[0461] Among them, the sensor module 280 may include a photoplethysmographic sensor (PPG), a pressure sensor, a capacitance sensor, an acceleration sensor, an ambient light sensor, a proximity light sensor, a touch sensor, etc.

[0462] The processor 210 may include one or more processing units. For example, the processor 210 may include an AP, a modem processor, a GPU, an ISP, a controller, a memory, a video codec, a DSP, a baseband processor, and / or an NPU. The different processing units may be independent devices or integrated into one or more processors. The controller may serve as the nerve center and command center of the wearable device. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0463] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0464] The processor 210 can execute the device control method provided in the embodiments of the present application. The processor 210 can include different components. For example, when the processor 210 integrates a CPU and a GPU, the CPU and GPU can cooperate to execute the device control method provided in the embodiments of the present application. For example, part of the algorithm in the device control method is executed by the CPU, and another part of the algorithm is executed by the GPU to achieve faster processing efficiency.

[0465] It should be understood that the terminal device and wearable device shown in the figure are only an example, and the terminal device and / or wearable device may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration.

[0466] In addition, those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0467] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0468] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0469] An embodiment of the present application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the steps in the above-mentioned method embodiments.

[0470] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. Computer-readable storage media may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0471] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0472] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0473] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0474] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A device control system, characterized in that: include: A first device and a second device, wherein a connection has been established between the first device and the second device, wherein the first device is a mobile phone and the second device is a wearable device; The first device performs owner identification processing in response to a first operation of a user triggering foreground running of a first application; When the first device identifies that the user is not the device owner, displaying a first interface on the second device, the first interface including a first window, the first window including a first prompt message and a first control, the first prompt message reminding that the first application is being used by a non-device owner; When the second device receives the user's operation on the first control, it controls the first device to lock the screen.

2. The device control system according to claim 1, characterized in that: The information transmission channel between the first device and the second device includes a first channel and a second channel; The first channel is an information transmission channel between an application in the first device and the second device, and the second channel is an information transmission channel between a system service in the first device and the second device; The application is located in the application program layer of the Android system, and the system service is located in the application program framework layer of the Android system.

3. The device control system according to claim 1, characterized in that: Before the second device displays the first interface, the first device sends a notification message to the second device through the second channel when it is identified that the user is not the owner of the device, and the notification message indicates that the first application is being used by a non-owner.

4. The device control system according to any one of claims 1 to 3, characterized in that: The controlling the first device to lock the screen includes: In response to a user operation on the first control, the second device sends a lock instruction to the first device through a second channel, where the second channel is an encrypted transmission channel; The first device locks the electronic screen of the first device based on the locking instruction.

5. The device control system according to claim 1, characterized in that: A target application is installed in the first device, the target application includes a first setting interface, and the first setting interface includes a first switch control; The first device enables an application usage reminder function in response to a user's operation of turning on the first switch control; The first device performs an owner identification process in response to the user turning off the first switch control, and turns off the application usage reminder function when the user is identified as the owner; Among them, the first device performs the owner identification processing in response to the first operation triggered by the user to run the first application in the foreground, including: when the application usage reminder function is turned on, the first device performs the owner identification processing in response to the first operation triggered by the user to run the first application in the foreground.

6. The equipment control system according to claim 1, characterized in that: The first setting interface of the target application includes a preset application list, and the first application belongs to the preset application list; The first device performs owner identification processing in response to a user's deletion operation on an application in the preset application list, and when the user is identified as the owner, deletes the application targeted by the deletion operation from the preset application list.

7. The equipment control system according to claim 1, characterized in that: The first setting interface of the target application includes setting controls; The first device performs an owner identification process in response to the user's operation on the setting control, and displays a second interface when the user is identified as the owner, the second interface including a plurality of second switch controls; In response to the user turning on the second switch control, the first device adds the application corresponding to the operated second switch control to the preset application list; In response to the user's operation of closing the second switch control, the first device deletes the application corresponding to the operated second switch control from the preset application list.

8. The equipment control system according to claim 1, characterized in that: The first setting interface of the target application includes a record viewing control; The first device displays historical access records of non-machine owners accessing applications in a preset application list in response to a user operation on the record viewing control.

9. The device control system according to any one of claims 5 to 8, characterized in that: The target application is a sports and health application.

10. The equipment control system according to claim 1, characterized in that: The first window further includes a second control; When the second device receives a user operation on the second control, the first device runs the first application in the foreground; When the first device receives the first operation for the first application again within the preset effective time, the first device runs the first application in the foreground.

11. The equipment control system according to any one of claims 1 to 10, characterized in that: The first device performs owner identification processing, including: The first device collects user data, wherein the user data includes at least one of the following: a face image to be tested, breathing audio data to be tested, IMU data to be tested, and touch screen data to be tested; Based on the user data and the pre-recorded owner template, determine whether the user is the owner of the device. The owner template includes at least one of the following: a face template, a breathing sound feature template, and a cross-modal feature template. The cross-modal feature template is a comprehensive feature of the owner in multiple modalities.

12. The equipment control system according to claim 11, characterized in that: The first device determines whether the user is the owner of the device based on the user data and a pre-recorded owner template, including: When a face is present in the face image to be tested, the first device performs owner recognition based on the face image to be tested and the face template to obtain a face recognition score; When the face recognition score is greater than a preset face recognition lower threshold and less than a preset face recognition upper threshold, extracting the breath sound feature of the user to be tested from the breath audio data to be tested; The breath sound feature to be measured, the IMU data to be measured, the touch screen data to be measured, and the face recognition score are input into a multimodal fusion device owner identification model to obtain device owner indication information, where the device owner indication information indicates whether the user is the device owner.

13. The equipment control system according to claim 12, characterized in that: The data processing process of the multimodal fusion machine owner identification model includes: Encoding the IMU data to be measured, the touch screen data to be measured, and the breath sound feature to be measured through a cross-modal encoder to obtain a cross-modal feature to be measured; Determining a cross-modal similarity between the cross-modal feature to be measured and a cross-modal feature template in the machine master template, and determining a respiratory sound similarity between the respiratory sound feature to be measured and a respiratory sound feature template in the machine master template; Determining a multimodal recognition score based on the cross-modal similarity, the breathing sound similarity, and the face recognition score, wherein the multimodal recognition score is a comprehensive recognition result of the user in multiple modalities; Determine whether the user is the device owner based on the multimodal recognition score and a preset fusion threshold.

14. A device control method, characterized in that: Applied to a first device, the method includes: The first device performs owner identification processing in response to a first operation of a user triggering foreground running of a first application; When it is identified that the user is not the device owner, sending a notification message to the second device, the notification message indicating that the first application is being used by a non-device owner; In response to receiving the lock instruction returned by the second device in response to the notification information, the electronic screen is locked.

15. A device control method, characterized in that: Applied to the second device, the method includes: The second device receives notification information from the first device, where the notification information indicates that the first application on the first device is being used by a non-device owner; In response to the notification information, displaying a first interface, the first interface including a first window, the first window including a first prompt message and a first control, the first prompt message reminding that the first application is being used by a non-user; In response to a user operation on the first control, a lock instruction is sent to the first device, where the lock instruction instructs the first device to lock the screen.

16. A terminal device, characterized in that: The terminal device is a first device, and the terminal device includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the device control method according to claim 14 is implemented.

17. A wearable device, characterized in that: The wearable device is a second device, and the wearable device includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the device control method according to claim 15 is implemented.

Images