Service methods and systems based on multi-terminal integration

Abstract

This disclosure relates to a service method and system based on multi-terminal integration. In response to establishing a connection with an electronic device and satisfying a first condition, a first timer is started; in response to the first timer timeout and satisfying a second condition, pre-configured resources in the in-vehicle device are enabled to run some or all of the algorithm modules involved in the pre-configured application in the electronic device; a first instruction message is sent to the electronic device, the first instruction message instructing the electronic device to stop running some or all of the algorithm modules; the first condition includes at least one of the following: the door of the vehicle where the in-vehicle device is located is opened and closed; a user is detected in the vehicle seat; the second condition includes: while maintaining the connection with the electronic device, the door of the vehicle where the in-vehicle device is located is not opened and closed, and / or a user is detected in the vehicle seat. This can save the computing power of the electronic device.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle infotainment systems, and in particular to a service method and system based on multi-terminal integration. Background Technology

[0002] Smart glasses, smartphones, and car infotainment systems are all standalone smart devices. Smart glasses and smartphones are relatively sensitive to power consumption, and their computing power is generally not as high as that of car infotainment systems. Summary of the Invention

[0003] This disclosure provides a service method based on multi-terminal integration and a corresponding service system for executing these methods.

[0004] According to a first aspect of the present disclosure, a service method based on multi-terminal fusion is provided, applied to an in-vehicle device, comprising: in response to establishing a connection with an electronic device and satisfying a first condition, starting a first timer; in response to the first timer timeout and satisfying a second condition, enabling pre-configured resources in the in-vehicle device to run some or all algorithm modules involved in a pre-configured application in the electronic device; sending a first indication information to the electronic device, the first indication information being used to instruct the electronic device to stop running the some or all algorithm modules; the first condition includes at least one of the following conditions: the door of the vehicle where the in-vehicle device is located is opened and closed; a user is detected in the seat of the vehicle; the second condition includes: while maintaining the connection with the electronic device, the door of the vehicle where the in-vehicle device is located is not opened and closed, and / or a user is detected in the seat of the vehicle.

[0005] Optionally, before enabling pre-configured resources in the vehicle-mounted device to run some or all of the algorithm modules involved in the pre-configured application in the electronic device, the method further includes: receiving second indication information sent by the electronic device, the second indication information including the running status of the pre-configured application in the electronic device; enabling pre-configured resources in the vehicle-mounted device to run some or all of the algorithm modules involved in the pre-configured application in the electronic device includes: in response to the running status of the pre-configured application in the electronic device being not running, enabling the pre-configured resources to run some or all of the algorithm modules.

[0006] Optionally, enabling the pre-configured resources to run some or all of the algorithm modules includes: starting a second timer; executing a first switching process; the first switching process includes: creating a task instance on the vehicle-mounted device based on the pre-configured resources for running some or all of the algorithm modules, the task instance being used to execute the data processing algorithms represented by the some or all of the algorithm modules for application data; establishing a communication channel between the vehicle-mounted device and the electronic device for transmitting the application data; wherein sending a first indication information to the electronic device includes: sending the first indication information to the electronic device in response to the start of the second timer; the method further includes: in response to the second timer timeout and the first switching process not being completed, sending a third indication information to the electronic device, the third indication information being used to instruct the electronic device to resume running some or all of the algorithm modules; or sending the first indication information to the electronic device includes: in response to the completion of the creation of both the task instance and the communication channel, sending the first indication information to the electronic device.

[0007] Optionally, the method further includes: starting a third timer in response to a third condition being met; and stopping the operation of some or all of the algorithm modules via pre-configured resources in the in-vehicle device in response to the third timer timing out and a fourth condition being met, wherein the third condition includes at least one of the following: the connection between the in-vehicle device and the electronic device is interrupted; the user leaves the seat; the vehicle is turned off; or the door is opened and then closed; and the fourth condition includes at least one of the following: the connection between the in-vehicle device and the electronic device is interrupted; the user leaves the seat; the vehicle is turned off; or the door is not opened and then closed.

[0008] Optionally, stopping the operation of some or all algorithm modules through pre-configured resources in the vehicle-mounted device includes: starting a fourth timer in response to the timeout of the third timer and the fulfillment of a fourth condition; executing a second switching process; the second switching process includes: detecting whether an application related to some or all algorithm modules is running in the vehicle-mounted device, and if the application is running, interrupting the application; sending a fourth indication message to the electronic device, the fourth indication message being used to instruct the electronic device to resume running some or all algorithm modules; deleting the task instance created on the vehicle-mounted device; deleting the communication channel between the vehicle-mounted device and the electronic device used for transmitting application data; and forcibly executing the step of sending the fourth indication message to the electronic device in response to the timeout of the fourth timer and the incomplete execution of the second switching process.

[0009] According to a second aspect of the present disclosure, a multi-terminal fusion-based service method is provided, applied to an electronic device, comprising: establishing a connection with an in-vehicle device, the in-vehicle device having pre-configured resources for running some or all algorithm modules involved in a pre-configured application in the electronic device; and stopping the running of some or all algorithm modules involved in the pre-configured application in the electronic device in response to receiving a first indication information sent by the in-vehicle device, the first indication information being sent by the in-vehicle device after determining to run some or all algorithm modules involved in the pre-configured application.

[0010] Optionally, the method further includes: sending second indication information to the vehicle-mounted device in response to establishing a connection with the vehicle-mounted device; or sending the second indication information to the vehicle-mounted device in response to receiving a query request sent by the vehicle-mounted device, wherein the second indication information includes the operating status of the pre-configured application in the electronic device.

[0011] Optionally, the method further includes: in response to receiving a third or fourth instruction message sent by the vehicle-mounted device, resuming the operation of some or all of the algorithm modules.

[0012] According to a third aspect of the present disclosure, a multi-terminal converged service system is provided, comprising: an electronic device and an in-vehicle device, wherein the in-vehicle device is configured to: in response to establishing a connection with the electronic device and satisfying a first condition, start a first timer; in response to the first timer timeout and satisfying a second condition, enable pre-configured resources in the in-vehicle device to run some or all algorithm modules involved in a pre-configured application in the electronic device; send a first indication message to the electronic device, the first indication message being used to instruct the electronic device to stop running the some or all algorithm modules, wherein the first condition includes at least one of the following conditions: the door of the vehicle where the in-vehicle device is located is opened and closed; a user is detected in the seat of the vehicle; the second condition includes: while maintaining a connection with the electronic device, the door of the vehicle where the in-vehicle device is located is not opened and closed, and / or a user is detected in the seat of the vehicle; the electronic device is configured to: establish a connection with the in-vehicle device; and in response to receiving the first indication message sent by the in-vehicle device, stop running some or all algorithm modules involved in a pre-configured application in the electronic device.

[0013] According to a fourth aspect of the present disclosure, a computing device is provided, including: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in the first or second aspect above.

[0014] According to a fifth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first or second aspect above. Attached Figure Description

[0015] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0016] Figure 1 An exemplary flowchart of a multi-terminal convergence-based service method according to at least one embodiment of the present disclosure is shown.

[0017] Figure 2 An example is shown Figure 1 The service method shown may also include a flowchart of steps.

[0018] Figure 3 An exemplary flowchart of a multi-terminal convergence-based service method according to at least one embodiment of the present disclosure is shown.

[0019] Figure 4 This is an exemplary schematic diagram of the structure of a multi-terminal converged service system according to at least one embodiment of the present disclosure.

[0020] Figure 5 This is an exemplary schematic diagram illustrating state transitions according to at least one embodiment of the present disclosure.

[0021] Figure 6 This is an exemplary schematic diagram of a state machine according to at least one embodiment of the present disclosure.

[0022] Figure 7 A schematic diagram of the structure of a computing device according to at least one embodiment of the present disclosure is shown as an example. Detailed Implementation

[0023] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0024] Those skilled in the art will understand that the terms "first," "second," etc., in this disclosure are used to distinguish similar objects, rather than to describe a specific order or sequence, and do not have any additional limiting effect.

[0025] Smart glasses, mobile phones, and other electronic devices are quite sensitive to power consumption, and most of these devices do not have the computing power of a car's infotainment system. Therefore, if the computing overhead of some or all of the algorithm modules involved in the pre-configured applications of electronic devices can be offloaded (i.e. migrated) to the car's infotainment system after the electronic devices are installed in the car, it would not only save the power of the electronic devices but also allow the use of larger, more complex, and more effective algorithms with the powerful computing resources of the car's infotainment system, which would greatly improve the user experience.

[0026] In view of this, this disclosure proposes a service method based on multi-terminal integration, which can enable pre-configured resources in the vehicle to run some or all of the algorithm modules involved in the pre-configured application in the electronic device when a user carrying an electronic device is detected to have stably entered the vehicle, thereby saving the computing power of the electronic device.

[0027] The electronic devices described in this disclosure can be any device that is convenient for a user to carry (including wearable devices), such as, but not limited to, mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), Internet of Things (IoT) devices, and other terminal devices. This disclosure does not limit the specific type of electronic device.

[0028] For example, the electronic device may be a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Personal Digital Assistant (PDA) device, handheld device with wireless communication capabilities, wearable device, computing device or other processing device connected to a wireless modem, computer, laptop computer, handheld communication device, handheld computing device, and / or other devices for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G (The 5th Generation mobile communication technology) networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0029] As an example and not a limitation, when the electronic device is a wearable device, it can also be a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as gloves and watches equipped with near-field communication modules. Wearable devices are portable devices worn directly on the body or integrated into a user's clothing or accessories. By attaching to the user and using a pre-bound electronic card, they perform operations such as payment and authentication. Wearable devices are not merely hardware devices; they achieve powerful functions through software support, data interaction, and cloud interaction. Broadly defined, wearable smart devices include those with comprehensive functions, large size, and the ability to perform complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those focused on a specific application function and requiring the use of other devices such as smartphones, such as various smartwatches and smart bracelets with displays.

[0030] Figure 1 An exemplary flowchart of a multi-terminal convergence-based service method according to at least one embodiment of the present disclosure is shown. Figure 1 The method shown can be applied to in-vehicle devices. In-vehicle devices can refer to various devices and systems installed in a vehicle to improve vehicle performance, enhance the driving experience, ensure driving safety, or provide entertainment functions.

[0031] For example, in-vehicle equipment can refer to the vehicle's infotainment system. The vehicle's infotainment system refers to the various electronic devices and service systems inside a vehicle, including audio, navigation, communication, and infotainment functions. These devices and service systems are typically integrated and coordinated through the Electronic Control Unit (ECU) to provide a safe, comfortable, and convenient driving experience. The vehicle's infotainment system is the computing hub of the entire vehicle system; it has powerful computing capabilities and can centralize previously distributed sub-computing modules to handle most of the computing tasks.

[0032] See Figure 1 In step S110, in response to establishing a connection with the electronic device and satisfying the first condition, the first timer is started.

[0033] The connection established between in-vehicle equipment and electronic devices can be wireless. Wireless connections can include, but are not limited to, Bluetooth connections and Wi-Fi (Wireless Fidelity) connections.

[0034] The initial wireless connection between the in-vehicle device and the electronic device can be established manually by the user, for example, by manually registering the in-vehicle device to bind it to the electronic device. Afterward, the in-vehicle device can automatically establish a wireless connection with the electronic device when it approaches the vehicle.

[0035] The first condition can be considered as the determining factor for an entry into the vehicle event. The first condition may include at least one of the following: the door of the vehicle containing the in-vehicle device is opened and closed; a user is detected in the vehicle's seat. The first condition may also include the electronic device connecting to the Wi-Fi provided by the in-vehicle device. When the first condition includes multiple conditions, these conditions can be related by a logical AND operation. That is, the first condition is determined to be true only if all conditions included in the first condition are met. The door can be any door in the vehicle, such as the driver's side door, the front passenger side door, or the rear passenger side door. The seat can refer to any seat in the vehicle, such as the driver's seat, the front passenger seat, or the rear seats. Correspondingly, the user can refer to either the driver or a passenger.

[0036] In-vehicle devices can use door sensors to detect door status and determine whether the condition "the door of the vehicle where the device is located has been opened and closed" is met based on changes in door status. The device can also determine this condition based on information displayed on the dashboard regarding door status. Furthermore, the device can analyze collected audio data to determine if the condition "the door of the vehicle where the device is located has been opened and closed" is met. For example, it can determine whether the condition is met by detecting whether a door closing sound is detected within a preset time range after the door opening sound is detected.

[0037] In-vehicle equipment can determine whether the condition "a user is detected in a vehicle seat" is met based on sensors, image processing, and other methods. For example, pressure sensors installed under the seat or in the seat cushion can be used to determine whether a user is in the seat. Another example is the use of infrared sensors, whose detection range can be a portion of the area above the seat. When a passenger sits down, they block or reflect infrared radiation, thus triggering the infrared sensor. Yet another example is the use of in-vehicle cameras to capture images of the seating area, with image processing algorithms used to identify whether a user is in the seat.

[0038] In some implementations, it can be first determined whether the in-vehicle device has established a connection with the electronic device. If the connection is established, then the first condition can be determined. If the first condition is not met, the system can wait for a fixed period of time (e.g., 1 second) and then determine whether the first condition is met again. Alternatively, it can return to the previous step and determine whether the in-vehicle device has established a connection with the electronic device again, so that the first condition is determined again in response to a new triggering condition (i.e., determining that the in-vehicle device has established a connection with the electronic device again).

[0039] In step S120, in response to the first timer timeout and the second condition being met, the pre-configured resources in the vehicle device are enabled to run some or all of the algorithm modules involved in the pre-configured application in the electronic device.

[0040] The second condition can be considered as the criterion for determining whether the user did not leave the vehicle after the entry incident. The second condition may include: while maintaining a connection with the electronic device, the door of the vehicle where the onboard device is located was not opened and closed, and / or the user was detected in the vehicle's seat.

[0041] In-vehicle devices can use door sensors to detect door status and determine whether the condition "the door of the vehicle where the device is located has not been opened and closed" is true based on changes in door status. The device can also determine this condition based on information displayed on the dashboard regarding door status. Furthermore, the device can analyze collected audio data to determine whether the condition "the door of the vehicle where the device is located has not been opened and closed" is true.

[0042] The method for determining the condition "the vehicle's seat detected a user" can be found in the description above.

[0043] In some implementations, after the first timer is started, the second condition can be continuously or intermittently checked. If the second condition is not met at any time before the first timer expires, it can be assumed that the user entered the car and then left, meaning the user did not reliably enter the car. In other words, the first timer expiring and the second condition being met can mean that the second condition is always met from the start of the first timer until its expiration.

[0044] By starting a first timer and checking if the second condition is met, frequent boarding and alighting of the vehicle within a short period can be prevented. If the first timer expires but the second condition is still met, the user can be considered to have stably entered the vehicle. Computational power migration only occurs after confirming stable user entry into the vehicle, thus preventing meaningless migration caused by ping-pong entry and exit. The computational power migration mentioned here refers to transferring the computing power used to run algorithms from electronic devices to in-vehicle equipment.

[0045] If the second condition is not met at any time after the first timer starts but before it expires, the subsequent service process can be terminated. At this time, it is possible to re-detect whether the vehicle-mounted device and the electronic device have established a connection and whether the first condition is met, and determine whether to return to step S110 based on the detection results.

[0046] Pre-configured applications refer to applications installed on electronic devices that require computing resources to run specific algorithms during operation. In some implementations, pre-configured applications may be applications involving speech and semantic algorithms, such as voice assistants, translation, phone calls, and music playback. That is, some or all of the algorithm modules involved in a pre-configured application may include audio algorithm modules and / or semantic algorithm modules. Audio algorithm modules may include, but are not limited to, one or more of echo cancellation, speech enhancement, voice wake-up, noise reduction, silence detection, and offline recognition. Semantic algorithm modules may include, but are not limited to, one or more of semantic classification, semantic rejection, semantic arbitration, named entity recognition, natural language generation, and text-to-speech generation.

[0047] Activating pre-configured resources in an in-vehicle device to run some or all of the algorithm modules involved in pre-configured applications in an electronic device refers to migrating some or all of the algorithm modules that originally required the computing resources of the electronic device to the in-vehicle device, so as to utilize the pre-configured resources in the in-vehicle device to execute the some or all of the algorithm modules. By migrating some or all of the algorithm modules from the electronic device to the in-vehicle device, on the one hand, the resource consumption of the electronic device can be reduced, saving the power of the electronic device; on the other hand, the powerful computing resources of the in-vehicle device can be leveraged to improve processing efficiency.

[0048] Pre-configured resources may include computing power resources and / or algorithm resources in the in-vehicle device. In some embodiments, the pre-configured resources in the in-vehicle device may include algorithm resources that have the same function as some or all of the algorithm modules but are more complex and have better performance, such as larger, more complex, and better audio algorithms and / or semantic algorithms. In this way, the computing power overhead in electronic devices can be saved while improving the user experience.

[0049] In step S130, a first instruction message is sent to the electronic device, which is used to instruct the electronic device to stop running some or all of the algorithm modules.

[0050] Once the vehicle-mounted device determines some or all of the algorithm modules involved in running the pre-configured application, it can send a first instruction message to the electronic device to cause the electronic device to stop running some or all of the algorithm modules based on its own computing resources.

[0051] In some implementations, before activating pre-configured resources in the vehicle device to run some or all of the algorithm modules involved in the pre-configured application in the electronic device, the vehicle device may also receive second instruction information sent by the electronic device. The second instruction information includes the running status of the pre-configured application in the electronic device. The vehicle device may activate the pre-configured resources to run the partial or all algorithm modules in response to the pre-configured application's running status being "not running" in the electronic device. That is, the vehicle device may only activate the pre-configured resources to run the partial or all algorithm modules if the pre-configured application's running status in the electronic device is "not running". If the pre-configured application's running status in the electronic device is "running", the electronic device can still run the partial or all algorithm modules based on its own computing resources, and can wait for the application to finish before performing algorithm migration. By performing algorithm migration only when the pre-configured application's running status in the electronic device is "not running", complex development logic can be avoided, and adverse effects on users caused by performing algorithm migration during normal use of the pre-configured application can be avoided.

[0052] This disclosure aims to migrate some or all of the algorithm modules involved in a pre-configured application from the user's electronic device to the in-vehicle device when a stable entry into the vehicle is detected. The execution results of these algorithm modules can be sent to the electronic device, where the pre-configured application still provides the corresponding service based on the results. Alternatively, the execution results of these algorithm modules can also be sent to the relevant application in the in-vehicle device, which then provides the corresponding service based on the execution results. In other words, the pre-configured application can continue to run in the electronic device, or it can stop running, and the in-vehicle device will call the relevant application (e.g., an application of the same type as the pre-configured application) to provide the corresponding service based on the execution results of the algorithm modules.

[0053] The following is an illustrative example of the specific process of enabling pre-configured resources to run some or all of the algorithm modules.

[0054] A second timer can be started and the first switching process can be executed.

[0055] The first switching process is used to characterize the operational procedures that need to be performed in order to enable the pre-configured resources in the on-board equipment to run some or all of the algorithm modules.

[0056] The first switching process may include: creating task instances on the in-vehicle device based on pre-configured resources to run some or all of the algorithm modules; and establishing a communication channel between the in-vehicle device and the electronic device for transmitting application data. The creation of task instances and the creation of the communication channel can be performed sequentially, such as in parallel.

[0057] A task instance is used to execute some or all of the data processing algorithms represented by the algorithm modules for application data. Taking audio algorithms or semantic algorithms as an example, instances of speech and semantic algorithm modules such as echo cancellation modules, speech enhancement modules, and wake-up modules can be created. Each algorithm can correspond to one task instance, or several algorithms can be in one task instance. The specific task instances can be created according to business requirements.

[0058] Application data can be considered as the processing object of the data processing algorithms represented by some or all of the algorithm modules. Taking audio algorithms or semantic algorithms as an example, where some or all of the algorithm modules represent data processing algorithms, application data can refer to the raw signals (such as raw audio data) collected by the electronic device through a sound sensor (such as a microphone). A communication channel (such as a Bluetooth voice channel) can be established based on the connection between the in-vehicle device and the electronic device. The algorithm resources required to create the task instance can be pre-downloaded and installed. For example, this can be done when the connection with the electronic device is established for the first time, and the algorithm resources can be updated as needed thereafter. In addition, the electronic device also needs to pre-download and install algorithm resources, which can also be done when the connection with the in-vehicle device is established for the first time, and can be updated as needed thereafter.

[0059] Once the task instance and communication channel are created, i.e. the first switching process is completed, the vehicle system is ready to take over the operation of some or all of the algorithm modules in the electronic device. The system can put some or all of the algorithm modules in the electronic device into hibernation or unload them to save power.

[0060] In some implementations, a first instruction message can be sent to the electronic device before the first switching process is completed. For example, the in-vehicle device can send the first instruction message to the electronic device in response to the start of a second timer. At this time, the electronic device suspends the provision of algorithm services (e.g., voice and semantic services) related to some or all of the algorithm modules during the time period of executing the first switching process. The in-vehicle device can also send a third instruction message to the electronic device in response to the second timer expires and the first switching process not being completed. The third instruction message is used to instruct the electronic device to resume running some or all of the algorithm modules. Thus, the second timer enables the continued use of the computing resources in the electronic device to provide algorithm services even if the first switching process cannot be completed after the timeout. In response to the second timer expires and the first switching process not being completed, the created parts during the execution of the first switching process by the in-vehicle device can also be deleted, for example, the created task instances and / or communication channels can be deleted.

[0061] In some implementations, the first instruction information can be sent to the electronic device after the first switching process has been completed. For example, the vehicle-mounted device can send the first instruction information to the electronic device in response to the completion of both the task instance and the communication channel.

[0062] Figure 2 An example is shown Figure 1 The service method shown may also include a flowchart of steps.

[0063] See Figure 2 In step S140, in response to the fulfillment of the third condition, the third timer is started.

[0064] The third condition can be considered as the criterion for determining an event of leaving the vehicle. The third condition may include at least one of the following: interruption of the connection between the in-vehicle equipment and electronic devices; the user leaving the seat; the vehicle being turned off; or the vehicle door being opened and then closed. When the third condition includes multiple of the above conditions, these conditions can be related by a logical OR. That is, if any one of the conditions included in the third condition is detected as true, the third condition can be determined to be true.

[0065] In step S150, in response to the third timer timing out and the fourth condition being met, the operation of some or all algorithm modules via the pre-configured resources in the vehicle-mounted equipment is stopped.

[0066] The fourth condition can be considered as a criterion for determining whether the user has re-entered the vehicle after the event of leaving the vehicle. The fourth condition may include at least one of the following: the connection between the in-vehicle equipment and electronic devices is interrupted; the user leaves their seat; the vehicle is turned off; or the door is closed without being opened. When the fourth condition includes multiple of the above conditions, these conditions can be logically ORed. That is, the fourth condition is determined to be true if any one of the conditions included in the fourth condition is detected to be true.

[0067] In some implementations, after the third timer is started, the fourth condition can be continuously or intermittently checked. If the fourth condition is not met at any time before the third timer expires, it can be assumed that the user left the car and then re-entered, meaning the user did not stably leave the car. In other words, the third timer expiring while the fourth condition is met can mean that the fourth condition is always met from the start of the third timer until its expiration.

[0068] By activating a third timer and checking if the fourth condition is met, frequent getting on and off the vehicle within a short period can be prevented. If the third timer expires but the fourth condition is still met, the user can be considered to have stably left the vehicle. Computational power migration only occurs after confirming the user's stable departure, thus avoiding meaningless migrations caused by ping-pong entry and exit from the vehicle. The computational power migration mentioned here refers to transferring the computing power used to run algorithms from the onboard equipment back to the electronic devices.

[0069] If the fourth condition is not met at any time after the third timer starts but before it expires, the computing power migration can be skipped, that is, step S150 can be skipped and some or all of the algorithm modules can still run through the pre-configured resources in the vehicle equipment.

[0070] The following is an illustrative example of the specific process for stopping the operation of some or all of the algorithm modules using pre-configured resources.

[0071] In response to the expiration of the third timer and the fulfillment of the fourth condition, the fourth timer can be started, and the second switching procedure can be executed. The second switching procedure is used to characterize the operational process required to stop the operation of some or all of the algorithm modules through the pre-configured resources in the on-board equipment.

[0072] The second switching process may include: detecting whether an application related to some or all of the algorithm modules is running in the vehicle-mounted device; if the application is running, interrupting the application; sending a fourth instruction message to the electronic device, which instructs the electronic device to resume running some or all of the algorithm modules; deleting the task instance created on the vehicle-mounted device; and deleting the communication channel between the vehicle-mounted device and the electronic device used to transmit application data.

[0073] For example, if music is detected playing on the in-vehicle device, playback will be stopped; or if a translation application is detected running on the in-vehicle device, the user will be prompted to launch the translation application again on their electronic device.

[0074] After sending the fourth instruction message to the electronic device, it can be assumed that the electronic device has taken over the algorithm services related to some or all of the algorithm modules. Therefore, after sending the fourth instruction message to the electronic device, the task instance and communication channel created on the in-vehicle device can be deleted. Deleting the task instance and deleting the communication channel can be performed in parallel without any order.

[0075] In some implementations, the engine is forcibly shut down after the user leaves the vehicle, at which point all applications on the in-vehicle device are forcibly closed, thus eliminating the need to delete task instances and communication channels already created on the in-vehicle device. However, for vehicles where the in-vehicle device remains powered on after the user leaves (such as new energy vehicles), the second handover process includes deleting task instances and communication channels already created on the in-vehicle device.

[0076] In response to the fourth timer timing out and the second handover process not being completed, the step of sending a fourth instruction message to the electronic device can be forcibly executed. Thus, the fourth timer can prevent a handover failure from stalling at a certain stage.

[0077] Figure 3 An exemplary flowchart of a multi-terminal convergence-based service method according to at least one embodiment of the present disclosure is shown. Figure 3 The method shown can be applied to electronic devices.

[0078] See Figure 3 In step S310, a connection is established with the vehicle-mounted equipment.

[0079] The in-vehicle device has pre-configured resources for running some or all of the algorithm modules involved in pre-configured applications in the electronic device. The connection established between the electronic device and the in-vehicle device can be a wireless connection (such as a Bluetooth connection).

[0080] The initial wireless connection between the electronic device and the in-vehicle device can be established manually by the user, for example, by manually registering the electronic device to bind it to the in-vehicle device. Afterward, the electronic device can automatically establish a wireless connection with the in-vehicle device when it approaches the vehicle.

[0081] In step S320, in response to receiving the first instruction information sent by the vehicle-mounted device, the operation of some or all of the algorithm modules involved in the pre-configured application in the electronic device is stopped.

[0082] The first instruction message is sent by the onboard equipment after it has determined some or all of the algorithm modules involved in the pre-configured application being run.

[0083] The electronic device can proactively send a second instruction to the in-vehicle device in response to establishing a connection. The second instruction includes the operating status of a pre-configured application within the electronic device. The electronic device can also send the second instruction to the in-vehicle device in response to receiving a query request from the in-vehicle device.

[0084] The electronic device can also resume operation of some or all of the algorithm modules in response to receiving a third or fourth instruction from the in-vehicle device. The third instruction is sent by the in-vehicle device to the electronic device if the algorithm migration process (i.e., the first switching process) times out and is not completed, instructing the electronic device to resume operation of some or all of the algorithm modules. The fourth instruction is sent by the in-vehicle device after determining that it will no longer run some or all of the algorithm modules involved in the pre-configured application, such as during the execution of the second switching process, instructing the electronic device to resume operation of some or all of the algorithm modules.

[0085] This disclosure also proposes a service system based on multi-terminal integration.

[0086] The service system may include electronic devices and in-vehicle equipment.

[0087] The vehicle-mounted equipment can be configured to perform the above-mentioned combination Figure 1 , Figure 2 The described steps and flow. Electronic devices can be configured to perform the above-described steps. Figure 3 The described steps and procedures.

[0088] For details on the specific operations that can be performed by electronic and in-vehicle devices, please refer to the relevant descriptions above.

[0089] Figure 4 This is an exemplary schematic diagram of the structure of a multi-terminal converged service system according to at least one embodiment of the present disclosure. Figure 4 As shown, the service system can include glasses, mobile phones, and car infotainment systems.

[0090] Glasses (such as smart glasses) and mobile phones are equivalent to the electronic devices mentioned above. Car infotainment systems are equivalent to the in-vehicle devices mentioned above. Some or all of the algorithm modules involved in pre-configured applications in mobile phones and glasses can be migrated to car infotainment systems.

[0091] Mobile phones, glasses, and car infotainment systems can connect to each other via Bluetooth.

[0092] Resources (such as backend service resources and algorithm resources) required by mobile phones, glasses, and car infotainment systems to implement the multi-terminal integrated service method can be pre-installed. Backend service resources refer to the program resources required by mobile phones, glasses, and car infotainment systems to execute the service method disclosed herein. Algorithm resources refer to the algorithm modules that need to be migrated.

[0093] Both mobile phones and car infotainment systems can connect to the cloud. For example, mobile phones and car infotainment systems can wirelessly connect to the cloud via 4G (4th Generation mobile communication technology) and 5G, respectively. Mobile phones, glasses, and car infotainment systems can obtain any resources they need from the cloud. Specifically, resources needed by the glasses can be obtained from the cloud by the mobile phone and sent to the glasses via Bluetooth.

[0094] exist Figure 4 In the illustrated embodiment, this disclosure can be implemented as a voice assistant that integrates glasses, a mobile phone, and a car infotainment system. This voice assistant leverages the powerful computing resources of the car infotainment system to migrate the voice algorithm modules from the glasses and mobile phone to the car infotainment system, saving battery power on the glasses and mobile phone, and utilizing the car infotainment system's large computing power and advanced models to improve the voice interaction experience.

[0095] Glasses, mobile phones, and car infotainment systems differ in their computing power, voice capabilities, screen sizes, and speaker configurations. Glasses are highly sensitive to power consumption and have limited computing power, with audio processing performance also constrained by available resources. However, due to their close proximity to the user, the audio data collected by their microphones is of high quality. Similarly, mobile phones are also power-sensitive and have limited computing power, but they are easy for users to operate and suitable for information distribution and security authentication. Car infotainment systems, on the other hand, possess powerful computing capabilities, providing strong audio processing performance. Therefore, the advantages of glasses, mobile phones, and car infotainment systems can be combined to provide users with an exceptional audio experience. For example, audio data collected by the glasses' microphone can be transmitted to a mobile phone, which in turn transmits it to the car infotainment system, where the system's computing resources are utilized for processing. The final processed results can then be sent to relevant applications on the glasses, mobile phone, or car infotainment system to provide application services to the user.

[0096] Figure 5 This is an exemplary schematic diagram illustrating state transitions according to at least one embodiment of the present disclosure.

[0097] like Figure 5 As shown, this disclosure involves the following four states: outside the vehicle, inside the vehicle, from outside the vehicle to inside the vehicle, and from inside the vehicle to outside the vehicle. "Outside the vehicle" and "inside the vehicle" are stable states, that is, states with a relatively long duration. "From outside the vehicle to inside the vehicle" and "from inside the vehicle to outside the vehicle" are unstable states, that is, states with a relatively short duration.

[0098] Figure 6 This is an exemplary schematic diagram of a state machine according to at least one embodiment of the present disclosure.

[0099] like Figure 6 As shown, this disclosure sets four states, namely "outside the vehicle", "inside the vehicle", "outside the vehicle to inside the vehicle" and "inside the vehicle to outside the vehicle", based on the switching of the two business processes of "entering the vehicle from outside the vehicle" and "moving from inside the vehicle to outside the vehicle", and realizes the switching between states by the change of the event-driven state machine.

[0100] The following explanation uses the state machine of an in-vehicle device as an example. It should be understood that in some implementations, the mobile phone, glasses, and in-vehicle device can all maintain a state machine. Furthermore, the mobile phone can be responsible for synchronizing the states of these three devices, ensuring that their current states are identical. For example, the state of the glasses or in-vehicle device can be sent to the mobile phone in real time. The mobile phone then determines whether the current states of the three devices are consistent based on its own data and the collected data. If they are inconsistent, it sends a state correction notification to the corresponding device based on pre-set correction logic to ensure that the current states of the three devices are consistent. In addition, the judgment conditions for received events by the mobile phone, glasses, and in-vehicle device in their current states can be different. Specifically, corresponding judgment conditions can be set for each event based on the capabilities of the device itself. For example, the mobile phone and glasses can simply detect entering or leaving the car based on whether a connection has been established with the in-vehicle device. As another example, when an app that can query vehicle status is installed on the mobile phone, the entry and exit events can also be detected in conjunction with information from the app.

[0101] When the current state is "outside the vehicle", the acceptable event is the entry into the vehicle event.

[0102] The criteria for determining an entry into the car event can be the aforementioned "establishing a connection with an electronic device (such as glasses and / or a mobile phone) and satisfying the first condition". In other words, the in-vehicle device can issue an entry into the car event when it detects that a connection has been established with an electronic device and the first condition has been met, and then execute the action "starting timer T1", switching the current state "outside the car" to the next state "from outside the car to inside the car".

[0103] The purpose of timer T1 is to prevent users from opening the car door, entering, and then exiting, thus avoiding the ping-pong effect. After timer T1 is started, it continuously (e.g., in real-time or intermittently) detects whether the user has left the car. If the user is detected leaving the car, timer T1 stops counting. Timer T1 corresponds to the first timer mentioned above. Timer T1 timeout indicates that the user has not left the car during the period from the start of timer T1 to the timer T1 timeout (i.e., the second condition is always true). Therefore, timer T1 timeout can be considered a stable entry into the car.

[0104] When the current state is "from outside the vehicle to inside the vehicle", three events can be accepted: T1 timeout, successful handover, and T2 timeout.

[0105] The T1 timeout event indicates that the user has successfully entered the vehicle. Therefore, in response to the T1 timeout event, the action of "starting the switch from outside to inside the vehicle and activating the switch timeout timer T2" can be executed without changing the current state. The switch from outside to inside the vehicle corresponds to the first switch procedure described above. Timer T2 is used to measure the switch time and corresponds to the second timer described above. If T2 times out, the switch is considered to have timed out.

[0106] A successful switch event indicates that the transition from outside to inside the vehicle has been completed. In response to this event, the action of "stopping timer T2" can be executed, and the current state "from outside to inside" can be switched to the next state "inside". Stopping timer T2 means that timer T2 stops counting.

[0107] A T2 timeout event indicates that the handover has not been completed for an extended period, suggesting that the handover has encountered a problem and failed. Therefore, in response to a T2 timeout event, a "handover rollback" action can be performed, switching the current state from "outside the vehicle to inside the vehicle" back to "outside the vehicle." Handover rollback means that the onboard equipment does not bear the computational overhead of some or all of the algorithm modules involved in the pre-configured applications in the electronic device, but the electronic device still runs those some or all algorithm modules.

[0108] When the current state is "inside the car", two events are acceptable: leaving the car and T3 timeout. The leaving the car event indicates that the user's action of moving from inside the car to outside the car has been detected. In response to the leaving the car event, the action of "starting timer T3" can be executed without changing the current state.

[0109] The purpose of timer T3 is to prevent the user from opening the car door, exiting, and then re-entering, thus avoiding the ping-pong effect. After timer T3 is started, it can continuously (e.g., in real-time or intermittently) detect whether the user has left the car. If it detects that the user has not left the car, timer T3 can stop counting. Timer T3 corresponds to the third timer mentioned above. Timer T3 timeout means that the user has not entered the car during the period from the start of timer T3 to the timer T3 timeout (i.e., the fourth condition is always true). Therefore, timer T3 timeout can be considered as the user having stably left the car.

[0110] The T3 timeout event signifies that the user has safely exited the car, similar in function to T1, preventing ping-pong entry and exit. In response to the T3 timeout event, the action of "starting the switch from inside the car to outside, activating the switch timeout timer T4" can be executed, and the current state "inside the car" can be switched to the next state "inside to outside the car". The switch from inside the car to outside the car corresponds to the second switch process described above. Timer T4 is used to measure the switch time, corresponding to the fourth timer described above. If timer T4 times out, the switch is considered to have timed out.

[0111] When the current state is "inside the vehicle to outside the vehicle", two events are acceptable: T4 timeout or successful switch.

[0112] The T4 timeout event indicates that the handover has not been completed for an extended period. To prevent handover failure, the process remains stalled at a certain stage. In response to the T4 timeout event, a "forced handover" action can be performed, switching the current state from "inside the vehicle to outside the vehicle" to the next state "outside the vehicle." A forced handover may refer to skipping the current step and sending information to the electronic device instructing it to resume operation of some or all of the algorithm modules (i.e., the fourth instruction information).

[0113] A successful switch event indicates that the transition from inside the vehicle to outside has been completed. In response to this event, the action of "stopping timer T4" can be executed, and the current state "from inside to outside" can be switched to the next state "outside". Stopping timer T4 means that timer T4 stops counting.

[0114] T1 and T3 are start timers used to ensure a smooth entry or exit from the car. The timeout duration for T1 and T3 can usually be set to a few seconds, such as 4 seconds. T2 and T4 are switch timers. The timeout duration for T2 and T4 can be set according to the actual software switch duration. Generally, 2 seconds is sufficient to complete the entire switch process.

[0115] Figure 7 A schematic diagram of the structure of a computing device according to at least one embodiment of the present disclosure is shown as an example.

[0116] See Figure 7 The computing device 700 includes a memory 710 and a processor 720.

[0117] Processor 720 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 720 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 720 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0118] Memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 710 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0119] The memory 710 stores executable code, which, when processed by the processor 720, enables the processor 720 to execute the multi-terminal converged service method described above.

[0120] The service method, system and device based on multi-terminal convergence according to this disclosure have been described in detail above with reference to the accompanying drawings.

[0121] Furthermore, the method according to this disclosure can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of this disclosure.

[0122] Alternatively, this disclosure may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the method described above according to this disclosure.

[0123] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A service method based on multi-terminal integration, applied to in-vehicle equipment, comprising: In response to establishing a connection with an electronic device and satisfying a first condition, a first timer is started; In response to the first timer timeout and the second condition being met, pre-configured resources in the vehicle device are enabled to run some or all of the algorithm modules involved in the pre-configured application in the electronic device. Send a first instruction message to the electronic device, the first instruction message being used to instruct the electronic device to stop running some or all of the algorithm modules; The first condition includes at least one of the following: the door of the vehicle containing the on-board device is opened and closed; a user is detected in the seat of the vehicle. The second condition includes: while maintaining a connection with the electronic device, the door of the vehicle in which the in-vehicle device is located is not opened and closed, and / or the user is detected in the seat of the vehicle.

2. The method according to claim 1, before enabling pre-configured resources in the vehicle-mounted device to run some or all of the algorithm modules involved in the pre-configured application in the electronic device, the method further includes: Receive a second indication information sent by the electronic device, the second indication information including the operating status of the pre-configured application in the electronic device; Enabling pre-configured resources in the vehicle-mounted device to run some or all of the algorithm modules involved in a pre-configured application in the electronic device includes: in response to the pre-configured application being in a non-running state in the electronic device, enabling the pre-configured resources to run some or all of the algorithm modules.

3. The method according to claim 2, wherein enabling the pre-configured resources to run some or all of the algorithm modules includes: Start the second timer; Execute the first handover process; The first switching process includes: creating a task instance on the vehicle-mounted device based on the pre-configured resources for running some or all of the algorithm modules, wherein the task instance is used to execute the data processing algorithms represented by some or all of the algorithm modules for application data; and establishing a communication channel between the vehicle-mounted device and the electronic device for transmitting the application data. The method of sending a first indication message to the electronic device includes: sending the first indication message to the electronic device in response to the start of the second timer; the method further includes: sending a third indication message to the electronic device in response to the timeout of the second timer and the incomplete execution of the first switching process, the third indication message being used to instruct the electronic device to resume running some or all of the algorithm modules; or Sending a first instruction message to the electronic device includes: in response to the completion of the creation of both the task instance and the communication channel, sending the first instruction message to the electronic device.

4. The method according to claim 1, further comprising: In response to the fulfillment of the third condition, the third timer is started; In response to the expiration of the third timer and the fulfillment of the fourth condition, the operation of some or all of the algorithm modules via the pre-configured resources in the on-board equipment is stopped. The third condition includes at least one of the following: the connection between the in-vehicle device and the electronic device is interrupted; the user leaves the seat; the vehicle is turned off; the vehicle door is opened and then closed. The fourth condition includes at least one of the following: the connection between the in-vehicle device and the electronic device is interrupted; the user leaves the seat; the vehicle is turned off; the door is closed without being opened.

5. The method according to claim 4, stopping the operation of some or all of the algorithm modules through pre-configured resources in the vehicle-mounted device, comprising: In response to the third timer timeout and the fourth condition being met, the fourth timer is started; Execute the second handover process; The second switching process includes: detecting whether an application related to some or all of the algorithm modules is running in the vehicle-mounted device; if the application is running, interrupting the application; sending a fourth instruction message to the electronic device, the fourth instruction message being used to instruct the electronic device to resume running some or all of the algorithm modules; deleting the task instance created on the vehicle-mounted device; and deleting the communication channel between the vehicle-mounted device and the electronic device used for transmitting application data. In response to the expiration of the fourth timer and the incomplete execution of the second switching process, the step of sending the fourth indication information to the electronic device is forcibly executed.

6. A service method based on multi-terminal convergence, applied to electronic devices, comprising: A connection is established with an in-vehicle device, which has pre-configured resources for running some or all of the algorithm modules involved in a pre-configured application in the electronic device; In response to receiving a first instruction message sent by the vehicle-mounted device, the operation of some or all algorithm modules involved in the pre-configured application in the electronic device is stopped. The first instruction message is sent by the vehicle-mounted device after determining that some or all algorithm modules involved in the pre-configured application are to be run.

7. The method according to claim 6, further comprising: In response to establishing a connection with the vehicle-mounted device, a second instruction message is sent to the vehicle-mounted device; or In response to receiving a query request from the vehicle-mounted device, the system sends the second indication information to the vehicle-mounted device, the second indication information including the operating status of the pre-configured application in the electronic device.

8. The method according to claim 6, further comprising: In response to receiving a third or fourth instruction from the vehicle-mounted device, the operation of some or all of the algorithm modules is resumed.

9. A service system based on multi-terminal integration, comprising: Electronic devices and in-vehicle equipment, The vehicle-mounted equipment is configured as follows: In response to establishing a connection with an electronic device and satisfying a first condition, a first timer is started; In response to the first timer timeout and the second condition being met, pre-configured resources in the vehicle device are enabled to run some or all of the algorithm modules involved in the pre-configured application in the electronic device. Send a first instruction message to the electronic device, the first instruction message being used to instruct the electronic device to stop running some or all of the algorithm modules, wherein... The first condition includes at least one of the following: the door of the vehicle containing the on-board device is opened and closed; a user is detected in the seat of the vehicle. The second condition includes: while maintaining a connection with the electronic device, the door of the vehicle where the in-vehicle device is located has not been opened and closed, and / or the user is detected in the vehicle's seat. The electronic device is configured to: Establish a connection with the vehicle-mounted equipment; In response to receiving the first indication information sent by the vehicle-mounted device, the operation of some or all of the algorithm modules involved in the pre-configured application in the electronic device is stopped.

10. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 8.

11. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 8.

Images