Information processing method and device, master control equipment and controlled equipment

Abstract

The embodiment of the invention discloses an information processing method and device, master control equipment and controlled equipment, and the method comprises the steps that the controlled equipment obtains first information of a first user, and sends the first information to the master control equipment; the main control equipment receives first information from the controlled equipment; the main control equipment determines the electronic equipment with the function of executing the target service identified by the first information from the registered equipment group to obtain a candidate equipment group; the main control equipment acquires a user decision parameter of the first user and first state information in the state information set of the first candidate equipment; the main control device determines a target device from the candidate device group according to the user decision parameter and the state information set; and the main control equipment sends second information to the target equipment, wherein the second information is used for controlling the target equipment to execute the target service function. Therefore, the embodiment of the invention is beneficial to ensuring that the decided target equipment better meets the requirements or intentions of the user, and provides more accurate personalized decisions for the user and better meets the personalized requirements of the user.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an information processing method and apparatus, a main control device, and a controlled device. Background Art

[0002] In daily life, users can usually meet different usage needs through different service functions (such as video playback, music playback, screen projection, etc.) provided by various electronic devices. For example, use TV to watch news broadcasts, use speakers to play music, use mobile phones to make calls, etc. At the same time, users can also achieve dynamic interaction with electronic devices through various control methods such as voice and gestures. For example, users input voice or gestures to control mobile phones to play music.

[0003] However, with the rapid development of IoT software and hardware, the service functions and processing capabilities that electronic devices can provide are constantly enhanced, that is, the differences between different types of electronic devices are gradually reduced, resulting in the homogenization of service functions of different types of electronic devices. For example, there are small differences between smart TVs and smart phones in service functions such as video playback and music playback. Therefore, in scenarios where users control multiple electronic devices through device control commands such as voice and gestures, how multiple electronic devices can autonomously decide on the appropriate electronic device to perform the service functions required by the user according to the user's device control commands still requires further research. Summary of the invention

[0004] The embodiments of the present application provide an information processing method and apparatus, a master control device and a controlled device, in the hope of ensuring that the target device decided upon is more in line with the needs or intentions of the user, providing the user with more accurate personalized decisions and better meeting the personalized needs of the user.

[0005] In a first aspect, an embodiment of the present application provides an information processing method, which is applied to a master control device, and the method includes:

[0006] Receiving first information from a first user of a controlled device, wherein the controlled device is an electronic device in a registered device group determined in response to device wake-up information of the first user, wherein the registered device group is composed of electronic devices registered with the master control device;

[0007] Determining electronic devices capable of executing the target service function identified by the first information from the registered device group to obtain a candidate device group;

[0008] Acquire a user decision parameter of the first user and first state information of a first candidate device, where the first candidate device is a candidate device in the candidate device group, and the first state information is a state information in a state information set;

[0009] Determining a target device from the candidate device group according to the user decision parameter and the state information set;

[0010] Sending second information to the target device, where the second information is used to control the target device to execute the target service function.

[0011] In a second aspect, an embodiment of the present application provides an information processing method, which is applied to a controlled device, and the method includes:

[0012] Acquire first information of a first user, wherein the controlled device is an electronic device in a registered device group determined in response to device wake-up information of the first user, and the registered device group is composed of electronic devices registered with a master control device;

[0013] The first information is sent to the master control device, where the first information is used by the master control device to determine a target device that executes a target service function identified by the first information from the registered device group.

[0014] In a third aspect, an embodiment of the present application provides an information processing device, which is applied to a main control device; the device includes a processing unit and a communication unit, and the processing unit is used to:

[0015] receiving, by the communication unit, first information from a first user of a controlled device, the controlled device being an electronic device in a registered device group determined in response to device wake-up information of the first user, the registered device group consisting of electronic devices registered with the master control device;

[0016] Determining electronic devices capable of executing the target service function identified by the first information from the registered device group to obtain a candidate device group;

[0017] Acquire a user decision parameter of the first user and first state information of a first candidate device, where the first candidate device is a candidate device in the candidate device group, and the first state information is a state information in a state information set;

[0018] Determining a target device from the candidate device group according to the user decision parameter and the state information set;

[0019] The second information is sent to the target device through the communication unit, where the second information is used to control the target device to execute the target service function.

[0020] In a fourth aspect, an embodiment of the present application provides an information processing device, which is applied to a controlled device; the device includes a processing unit and a communication unit, and the processing unit is used to:

[0021] Acquire first information of a first user, wherein the controlled device is an electronic device in a registered device group determined in response to device wake-up information of the first user, and the registered device group is composed of electronic devices registered with a master control device;

[0022] The first information is sent to the main control device through the communication unit, and the first information is used by the main control device to determine a target device that executes the target service function identified by the first information from the registered device group.

[0023] In a fifth aspect, an embodiment of the present application provides a main control device, which includes an electronic device or a server; the main control device includes a processor, a memory and a communication interface, the memory stores one or more programs, and the one or more programs are executed by the processor, and the one or more programs are used to execute the instructions of the steps in the first aspect of the embodiment of the present application.

[0024] In the sixth aspect, an embodiment of the present application provides a controlled device, wherein the controlled device includes an electronic device; the main control device includes a processor, a memory and a communication interface, the memory stores one or more programs, and the one or more programs are executed by the processor, and the one or more programs are used to execute instructions of the steps in the second aspect of the embodiment of the present application.

[0025] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program is operable to enable a computer to execute part or all of the steps described in the first aspect or the second aspect of the embodiment of the present application.

[0026] In an eighth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0027] It can be seen that in the embodiment of the present application, first, the controlled device obtains the first information of the first user and sends the first information to the main control device; secondly, the main control device receives the first information from the controlled device, and determines, from the registered device group, an electronic device that has the target service function identified by the first information to obtain a candidate device group; thirdly, the main control device obtains the user decision parameters of the first user and the first status information of the first candidate device; finally, the main control device determines the target device from the candidate device group according to the user decision parameters and the status information set, and sends the second information to the target device. Since the embodiment of the present application first screens out a candidate device group capable of executing the target service function from the registered device group (i.e., the initial screening), and then screens out the target device that ultimately executes the target service function from the candidate device group (i.e., the second screening), in the initial screening, the main control device only needs to simply and preliminarily screen out the electronic devices capable of executing the target service function, while in the second screening, the main control device needs to make a joint decision based on the user decision parameters and state information set of the first user, which requires a relatively large amount of calculation. Therefore, by performing a second screening (i.e., secondary screening) on ​​the basis of the initial screening, it is beneficial to improve the efficiency of collaborative control between devices, ensure that the target device decided is more in line with the user's intention, and improve the accuracy of device decision. In addition, the embodiment of the present application makes personalized device decisions based on the user decision parameters and state information set of the first user, ensuring that the target device decided is more in line with the user's needs or intentions, providing users with more accurate personalized decisions and more personalized needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a schematic diagram of the architecture of a collaborative control system between devices provided in an embodiment of the present application;

[0030] Figure 2 This is a schematic diagram of a device wake-up process provided by an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of a process of determining a target device provided by an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of the hardware structure of a master control device or a controlled device provided in an embodiment of the present application;

[0033] Figure 5It is a schematic diagram of the software structure of a master control device or a controlled device provided in an embodiment of the present application;

[0034] Figure 6 It is a flowchart of an information processing method provided by an embodiment of the present application;

[0035] Figure 7 It is a structural diagram of a service function analysis model provided in an embodiment of the present application;

[0036] Figure 8 It is a structural schematic diagram of a hand key point provided in an embodiment of the present application;

[0037] Fig. 9 It is a structural schematic diagram of a hand key point provided in an embodiment of the present application;

[0038] Fig.10 is a structural schematic diagram of a target gesture type identified in gesture control information provided by an embodiment of the present application;

[0039] Fig.11 It is a flowchart of another information processing method provided in an embodiment of the present application;

[0040] Fig.12 It is a block diagram of the functional units of an information processing device provided in an embodiment of the present application;

[0041] Fig.13 It is a block diagram of the functional units of another information processing device provided in an embodiment of the present application;

[0042] Fig.14 It is a structural diagram of a main control device provided in an embodiment of the present application;

[0043] Fig.15 It is a structural schematic diagram of a controlled device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to help those skilled in the art better understand the technical solutions of the present application, the following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the description of the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0045] Before describing the technical solutions of the embodiments of the present application, the following first introduces the related concepts that may be involved in the present application, the collaborative control system between devices, the software and hardware structure of the devices, etc.

[0046] (1) Collaborative control system between devices

[0047] The collaborative control system between devices in the embodiment of the present application may include a master control device and a registered device group. The master control device and each electronic device in the registered device group may establish a communication link through a wireless network or a wired network. At the same time, each electronic device in the registered device group may also establish a communication link through a wireless network or a wired network; or, no communication link is established between each electronic device in the registered device group.

[0048] Specifically, wireless networks may include mobile cellular networks (such as the fifth generation 5G mobile communication network), satellite communication networks, the Internet of Things, wireless local area networks (WLAN), wide area networks (WAN), Bluetooth (BT), wireless fidelity (Wi-Fi), Zigbee, near field communication (NFC), ultra wide band (UWB), visible light communication (LiFi), infrared technology (IR), etc.; wired networks may be networks constructed using data cables such as high definition multimedia interface (HDMI) data cables and universal serial bus (USB) data cables.

[0049] Specifically, the electronic device of the embodiment of the present application may be a handheld device, a vehicle-mounted device, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, a projection device, a projector, or other devices connected to a wireless modem, or may be various specific forms of user equipment (UE), terminal device, smart phone, smart screen, smart TV, smart watch, laptop computer, smart speaker, camera, game controller, microphone, station (STA), access point (AP), mobile station (MS), personal digital assistant (PDA), personal computer (PC) or relay device, etc.

[0050] Furthermore, the electronic devices of the embodiments of the present application can be specifically divided into the following six categories: the first device type has basic voice input, voice output or voice recognition capabilities, and only serves as an input interface for the user's voice control information or device wake-up information, but does not execute or provide the service function identified by the voice control information (or voice control instructions, voice control commands); the second device type has basic gesture input, gesture output or gesture recognition capabilities, and only serves as an input interface for the user's gesture control information or device wake-up information, but does not execute or provide the service function identified by the gesture control information; the third device type has voice input, voice output, voice recognition and execution of voice The fourth device type has the capabilities of gesture input, gesture output, gesture recognition, and the execution of service functions identified by gesture control information, such as smartphones, cameras, etc. The fifth device type does not have the capabilities of voice input, voice output, or voice recognition, but only has the capabilities of executing service functions identified by voice control information, such as air conditioners, refrigerators, washing machines, etc. The sixth device type does not have the capabilities of gesture input, gesture output, or gesture recognition, but only has the capabilities of executing service functions identified by gesture control information, such as air conditioners, refrigerators, washing machines, etc. Therefore, the embodiments of the present application can involve various types of electronic devices.

[0051] Specifically, the master control device may be a device with functions such as device wake-up decision, device service capability decision, device collaborative control, voice recognition, gesture recognition, etc. The master control device may include an electronic device, a server, or a hardware module with the above functions. Among them, the server in the embodiment of the present application may be a cloud server, an Internet of Vehicles cloud platform, an Internet of Vehicles cloud server, an Internet of Things cloud server, an Internet of Things device, a data center network device, a cloud computing device, etc.

[0052] For example, taking the main control device as a server, see Figure 1 The collaborative control system 10 between devices may include a server 110 and a registered device group 120. The server 110 and each electronic device in the registered device group 120 may establish a communication link through a wireless network or a wired network. The registered device group 120 may include a mobile phone 1201, a smart TV 1202, a laptop computer 1203, a smart speaker 1204, a smart watch 1205, etc. At the same time, each electronic device in the registered device group 120 may communicate with each other through a wireless network or a wired network.

[0053] Furthermore, the master device may be a device that acquires, stores and manages device registration information of each electronic device in the registered device group, wherein the device registration information includes at least one of distance information, service function information, device identification information, historical usage record information, device location information, device status information, and service status suitability information. The relevant description of the device registration information will be described in detail below.

[0054] Furthermore, the master control device can obtain, determine or generate a control command for controlling the target device in the registered device group to execute the service function identified by the user's voice control information or gesture control information by using the stored device registration information. In addition, the master control device can also receive the execution result after the target device executes the service function according to the control command.

[0055] Specifically, the registered device group may be composed of electronic devices that register with the master device. The electronic device registers with the master device, which can be understood as the electronic device uploading its own device identification information (such as media access control MAC address, Internet protocol IP address, device serial number SN, etc.), its own service function information (such as the mobile phone has video playback, etc.), its own device status information (such as remaining power, etc.), its own service status suitability information (such as whether the mobile phone is suitable for performing other service functions when the mobile phone is performing video playback), and its own device wake-up information input by the user (or periodically upload, upload in response to the request of the registered device) to the master device, and the master device stores or manages it, thereby completing the registration of the electronic device.

[0056] Furthermore, the electronic device may upload the above information to the main control device to complete registration when establishing an initial communication connection with the main control device, or may upload the above information together with the device wake-up information input by the user to the main control device to complete registration when obtaining the device wake-up information input by the user, and there is no specific limitation on this.

[0057] Furthermore, the electronic device in the registered device group may be installed with an intelligent voice assistant or an intelligent gesture assistant, so that the electronic device can realize the voice control function or the gesture control function. Since the intelligent voice assistant or the intelligent gesture assistant in the electronic device is usually in a dormant state to save power, before the user needs to use the voice control function or the gesture control function of the electronic device, it is necessary to perform a wake-up operation (such as voice wake-up or gesture wake-up) on the intelligent voice assistant or the intelligent gesture assistant. Among them, the device wake-up information for waking up the intelligent voice assistant can be called a wake-up word (such as "Xiaobu Xiaobu"), and the device wake-up information for waking up the intelligent gesture assistant can be called a wake-up gesture (such as "OK" gesture). At the same time, the wake-up word or the wake-up gesture can be pre-registered or stored in the electronic device. It should be noted that waking up the intelligent voice assistant may refer to the electronic device responding to the wake-up word input by the user to start the intelligent voice assistant, and waking up the intelligent gesture assistant may refer to the electronic device responding to the wake-up gesture input by the user to start the intelligent gesture assistant. In addition, the above-mentioned voice control function may mean that after the intelligent voice assistant of the electronic device is started, the user inputs voice control information to the electronic device (such as a voice of "playing the news broadcast on TV"), thereby triggering the electronic device to automatically execute the service function identified by the voice control information. The above-mentioned gesture control function may mean that after the intelligent gesture assistant of the electronic device is started, the user inputs gesture operation information to the electronic device, thereby triggering the electronic device to automatically execute the service function identified by the gesture control information.

[0058] Furthermore, each electronic device in the registered device group can be associated with the user identification information of the same user. The user identification information may include at least one of physiological feature information (such as hand shape, fingerprint, voiceprint, iris, face, etc.) and user account information (such as user account number, etc.). For example, the user has logged in to each electronic device in the registered device group using the same user account; the user has used voice / face / fingerprint / gesture unlocking or verification functions to open each electronic device in the registered device group, etc.

[0059] Furthermore, the registered device group includes a controlled device (or called a wake-up device), and the controlled device may be an electronic device in the registered device group that is determined in response to the device wake-up information of the first user.

[0060] For example, the device wake-up process is described by taking the master device as a server and the registered device group including a smart speaker, a smart phone, and a smart TV as an example. Figure 2. First, the user inputs the device wake-up information around him, such as "Xiaobu Xiaobu"; secondly, the electronic device with voice input function (smart speaker, smart phone and smart TV) receives the device wake-up information; wherein, the electronic device is equipped with a smart voice assistant and is in a dormant state; thirdly, the electronic device matches the device wake-up information through the pre-stored wake-up word. If the match is successful, the signal strength of the device wake-up information received, its own service capability information, its own device identification information, etc. are uploaded to the server; then, the server receives the above information to complete the registration of the electronic device, and responds to the device wake-up information according to the preset wake-up rules (such as the device with the closest distance, the device with the most recent historical use time, the device with the highest historical use frequency, etc.) to determine the wake-up of the smart speaker, and send a control instruction to the smart speaker; finally, the smart speaker receives the control instruction, turns on its own voice smart assistant, and sends a prompt message to the user (such as "Here, master").

[0061] It should be noted that the server can determine the distance between the user and the electronic device through the signal strength of the device wake-up information in order to execute the device distance closest wake-up rule. The server can determine the historical usage time of the electronic device by the user through the device identification information of the electronic device in order to execute the device historical usage time latest wake-up rule. The server can determine the historical usage frequency of the electronic device by the user through the device identification information of the electronic device in order to execute the device historical usage frequency highest wake-up rule.

[0062] Further, the registered device group may include a target device, which may be one or more electronic devices in the registered device group for executing the service function identified by the voice control information or gesture control information input by the user. It is understandable that the target device may include the above-mentioned controlled device.

[0063] For example, the process of determining the target device is described by taking the master device as a server, the registered device group including smart speakers, smart phones and smart TVs, and the controlled device as a smart speaker as an example. Figure 3. First, the user inputs voice control information, such as "play news broadcast"; second, the smart speaker with the smart voice assistant turned on receives the voice control information and uploads the voice control information to the server; third, the server identifies the voice operation information to determine the target service function, such as content playback, and determines that the electronic device that executes the target service function is a smart TV through the stored service capability information, device status information, service status suitability information, historical usage record information, etc. of the registered electronic devices (smart speakers, smart phones, smart TVs); then, the server generates a control command, which is used to control the smart TV to execute the target service function, and sends the control command to the smart TV; finally, the server obtains the control command and executes it.

[0064] Combine the following Figure 4 The structure of the master control device or the controlled device (hereinafter referred to as the device) in the embodiment of the present application is described in detail. It can be understood that: Figure 4 The schematic structure does not constitute a specific limitation on the device. In other embodiments of the present application, the device may also include Figure 4 Indicates more or fewer components, or combines some components, or separates some components, or arranges components differently. Figure 4 The illustrated components may be implemented by hardware, software, or a combination of software and hardware.

[0065] See also Figure 4 The device may include a processor 410, an antenna 1, an antenna 2, a mobile communication module 420, a wireless communication module 430, an audio module 440, a sensor module 450, a display module 460, a camera module 470, a charging management module 480, an internal memory 4901 and an external memory interface 4902, etc.

[0066] Specifically, the processor 410 may include one or more processing units. For example, the processor 410 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a baseband processor and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0067] Furthermore, a memory may be provided in the processor 410 for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. The memory may store instructions or data that the processor 410 has just used or cyclically used. If the processor 410 needs to use the instruction or data again, it may be directly called from the memory, thereby avoiding repeated access, reducing the waiting time of the processor 410 and improving system efficiency.

[0068] Further, the processor 410 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0069] It should be noted that the wireless communication function can be implemented by antenna 1, antenna 2, mobile communication module 420, wireless communication module 430, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the device can be used to cover a single or multiple communication frequency bands. In addition, different antennas can also be reused to improve the utilization of the antenna. For example, antenna 1 is reused as a diversity antenna of a wireless local area network.

[0070] Specifically, the mobile communication module 420 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied on the device. The mobile communication module 420 can include at least one filter, a switch, a power amplifier, and a low noise amplifier (LNA).

[0071] Furthermore, the mobile communication module 420 can receive electromagnetic waves through the antenna 1, filter, amplify and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. In addition, the mobile communication module 420 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0072] Furthermore, at least some functional modules of the mobile communication module 420 may be disposed in the processor 410; or, at least some functional modules of the mobile communication module 420 may be disposed in the same device as some modules of the processor 410.

[0073] Specifically, the wireless communication module 430 can provide wireless communication solutions applied on the device, including Bluetooth (BT), wireless local area networks (WLAN), wireless fidelity (Wi-Fi) networks, near field communication (NFC), infrared technology (IR), etc.

[0074] Furthermore, the wireless communication module 430 may be one or more devices integrating at least one communication processing module. In addition, the wireless communication module 430 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 410. The wireless communication module 430 may also receive a signal to be sent from the processor 410, modulate and amplify the signal, and then convert it into an electromagnetic wave and radiate it out through the antenna 2.

[0075] It should be noted that the device can implement audio functions such as music / video playback and recording through the audio module 440, the speaker 4401, the receiver 4402, the microphone 4403, the headphone interface 4404 and the processor 410.

[0076] Specifically, the audio module 440 can be used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signal. In addition, the audio module 440 can also be used to encode and decode audio signals. In some possible examples, the audio module 440 can be arranged in the processor 410, or some functional modules of the audio module 440 can be arranged in the processor 410.

[0077] Specifically, the speaker 4401 can be used to convert the audio electrical signal into a sound signal. The device can listen to the sound played in the music / video, or listen to the hands-free call, etc. through the speaker 4401.

[0078] Specifically, the receiver 4402 can be used to convert the audio electrical signal into a sound signal. When the device receives a call or voice message, the voice can be received by placing the receiver 4402 close to the human ear.

[0079] Specifically, microphone 4403 can be used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching microphone 4403 with his mouth to input the sound signal into microphone 4403. In addition, the device can be provided with at least one microphone 4403. In some possible examples, the electronic device can be provided with two microphones 4403, which can realize noise reduction function in addition to collecting sound signals; in some possible examples, the device can also be provided with three, four or more microphones 4403, which can realize the collection of sound signals or noise reduction, and can also identify the sound source to realize directional recording function, etc., without specific limitation.

[0080] Specifically, the earphone jack 4404 can be used to connect a wired earphone. The earphone jack 4404 can be a USB interface 4803, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface, etc.

[0081] Specifically, the sensor module 450 may include an inertial sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, an ultra-wideband UWB sensor, a near-field communication NFC sensor, a laser sensor and / or a visible light sensor, etc.

[0082] It should be noted that the device can implement the display function through a GPU, a display module 460, and a processor 410. Among them, the GPU can be used to perform mathematical and geometric calculations and perform graphic rendering. In addition, the GPU can be a microprocessor for image processing and connect the display module 460 and the processor 410. The processor 410 may include one or more GPUs, which execute program instructions to generate or change display information.

[0083] Specifically, the display module 460 may be a display screen for displaying images, videos, and the like. The display module 460 may include a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a quantum dot light-emitting diode (QLED), and the like. In some possible examples, the device may include one or more display modules 460.

[0084] It should be noted that the device can realize the shooting function through ISP, DSP, camera module 470, video codec, GPU, display module 460 and processor 410. Among them, ISP can be used to process the data fed back by the camera module 470. For example, when taking a photo, the shutter is opened first, and then the light is transmitted to the camera photosensitive element through the lens to realize the conversion of the light signal into an electrical signal, and finally the electrical signal is transmitted to the ISP through the camera photosensitive element for processing to be converted into an image visible to the naked eye. In addition, ISP can also perform algorithm optimization on the noise, brightness and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some possible examples, ISP or DSP can be set in the camera module 470.

[0085] Specifically, the camera module 470 can be a camera, which is used to capture static / dynamic images or videos. Among them, the object generates an optical image through the lens and projects it onto the photosensitive element, and the photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some possible examples, the device may include one or more camera modules 470.

[0086] Specifically, the charging management module 480 can be used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some possible examples, the charging management module 480 can receive charging input from a wired charger through a USB interface 4803. In some possible examples, the charging management module 480 can receive wireless charging input through a wireless charging coil of the device. While the charging management module 480 is charging the battery 4801, it can also power the device through the power management module 4802.

[0087] It should be noted that the power management module 4802 can be used to connect the battery 4801, the charging management module 480 and the processor 410. The power management module 4802 receives input from the battery 4801 and / or the charging management module 480 to supply power to various modules and the processor 410 in the device.

[0088] Specifically, the power management module 4802 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some possible examples, the power management module 4802 can also be set in the processor 410; in some possible examples, the power management module 4802 and the charging management module 480 can also be set in the same device.

[0089] It should be noted that the internal memory 4901 can be used to store computer executable program codes, which include instructions. Among them, the processor 410 executes various functional applications and data processing of the device by running the instructions stored in the internal memory 4901. In some possible examples, the internal memory 4901 stores program codes for executing the technical solutions of the embodiments of the present application.

[0090] Specifically, the internal memory 4901 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (e.g., a sound playback function and an image playback function, etc.). The data storage area may store data created during the use of the device (e.g., audio data and a phone book, etc.). In addition, the internal memory 4901 may include a high-speed random access memory and may also include a non-volatile memory. For example, at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0091] Specifically, the external memory interface 4902 can be used to connect an external memory card, such as a micro SD card, to expand the storage capacity of the device. The external memory card communicates with the processor 410 through the external memory interface 4902 to implement a data storage function. For example, files such as music and videos are stored in the external memory card.

[0092] In the embodiment of the present application, the software system of the master device or the controlled device (hereinafter referred to as the device) can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture or a cloud architecture. The following embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the master device.

[0093] like Figure 5 The schematic diagram of the architecture of the hardware and software system of the Android system is shown. Among them, the internal memory 4901 can store the kernel layer 520, the system runtime layer 540, the application framework layer 560 and the application layer 580. Among them, the layers communicate with each other through the software interface, and the kernel layer 520, the system runtime layer 540 and the application framework layer 560 belong to the operating system space.

[0094] Specifically, the application layer 580 belongs to the user space, and at least one application (or simply referred to as "application") runs in the application layer 580. These applications can be native applications that come with the operating system or third-party applications developed by third-party developers. For example, the application layer 580 can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, smart gesture assistant, smart voice assistant, video, and short message.

[0095] It should be noted that the application framework layer 560 provides various application programming interfaces (APIs) and programming frameworks that may be used by applications that build the application layer, so that developers can also build their own applications by using these APIs, such as window managers, content providers, view systems, telephony managers, resource managers, notification managers, message managers, activity managers, package managers, location managers, and NFC services.

[0096] Specifically, the window manager can be used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0097] Specifically, content providers can be used to store and retrieve data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. In addition, content providers can allow applications to access the data of another application, such as a contact database, or share their own data.

[0098] Specifically, the view system includes visual controls. For example, controls for displaying text and controls for displaying pictures. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying pictures.

[0099] Specifically, the phone manager is used to provide communication functions of the electronic device, for example, management of call status (such as connected, hung up, etc.).

[0100] Specifically, the resource manager can provide various resources for the application, such as localized strings, icons, images, layout files, video files, etc.

[0101] Specifically, the notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text. In addition, notifications of applications running in the background can also be notifications that appear on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0102] Specifically, the message manager may be used to store the data of messages reported by each application and process the data reported by each application.

[0103] Specifically, the activity manager can be used to manage the application life cycle and provide a common navigation back function. In a possible example, the message manager can be a part of the notification manager.

[0104] It should be noted that the system runtime layer 540 provides major feature support for the Android system through some C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D drawing support, and the Webkit library provides browser kernel support. The system runtime layer 540 also provides an Android runtime library (AndroidRuntime), which mainly provides some core libraries that allow developers to use the Java language to write Android applications.

[0105] Specifically, the kernel layer 520 can provide underlying drivers for various hardware of the electronic device, such as display driver, audio driver, camera driver, Bluetooth driver, Wi-Fi driver, power management, NFC driver, UWB driver, etc.

[0106] With the development of Internet software and hardware, electronic devices with different functions surround users, such as mobile phones, tablets, smart speakers, electronic watches, etc. As a tool that connects multiple devices to interact with users, smart voice assistants can make the connection between devices and users closer, thus achieving the strategic goal of the integration of all things.

[0107] In the process of interaction between intelligent voice assistants and users, they usually face the problem of multi-device selection. For example, when a user sends a voice control message of "play music" to the intelligent voice assistant, electronic devices such as mobile phones, TVs or speakers can provide music playback service functions. In this scenario, the intelligent voice assistant should focus on the user's attention and deduce the intended device that the user needs to control through the current device status information, user habits and other conditions, so as to intelligently provide the required service to the user.

[0108] Combined with the above description, the following will introduce the execution steps of the information processing method from the perspective of method examples. Figure 6 . Figure 6 : is a flow chart of an information processing method provided by an embodiment of the present application, the method comprising:

[0109] S610: The controlled device obtains first information of a first user, and sends the first information to the control device.

[0110] The controlled device may be an electronic device in the registered device group determined in response to the device wake-up information of the first user; the registered device group may be composed of electronic devices registered with the master control device.

[0111] The first information may be a device control command. The first information may be used by the master control device to determine a target device that executes a target service function identified by the first information from a registered device group.

[0112] Specifically, the first information may include voice control information input by the first user or gesture control information input by the first user.

[0113] It should be noted that, first, the controlled device can obtain voice control information (such as voice signals or readable text information converted from voice signals, etc.) input by the first user through a microphone; the controlled device can obtain gesture control information input by the first user through a camera module.

[0114] Secondly, if the voice control signal is a voice signal (represented in the form of an analog signal or a digital signal, etc.), the controlled device can first convert the sound received by the microphone into an acoustic signal, then remove the relevant environmental noise from the acoustic signal to obtain the voice signal, and upload the voice signal to the main control device. Alternatively, if the voice control information is an acoustic signal, the controlled device directly uploads the acoustic signal to the main control device, and then the main control device performs environmental noise and other processing, and there is no specific restriction on this.

[0115] Again, if the voice control signal is readable text information converted from a voice signal, the controlled device can convert the voice signal into computer-readable text through technologies such as automatic speech recognition (ASR), and then upload the computer-readable text to the main control device. There is no specific restriction on this.

[0116] Next, if the gesture control information is a video frame (or an image frame sequence, a frame image, an image frame, etc.), the controlled device can collect video frames (or an image frame sequence, a frame image, etc.) of the user's hand (one hand or both hands) through the camera module, and upload the video frame (or an image frame sequence, a frame image, etc.) to the main control device, without any specific restrictions.

[0117] Finally, if the gesture control information is hand feature information, the controlled device can first collect video frames (or image frame sequences, frame images, image frames, etc.) of the user's hands (one hand or both hands) through the camera module, and then perform feature extraction through image processing, image recognition and other technologies to obtain hand feature information, and upload the hand feature information to the main control device. There is no specific restriction on this.

[0118] S620: The master control device receives first information from the controlled device.

[0119] It should be noted that, in combination with the above description, first, the master control device of the embodiment of the present application can obtain, store and manage the device registration information of each electronic device in the registered device group. Among them, the device registration information of the electronic device may include at least one of distance information (such as the distance between the user and the electronic device), service function information (such as the electronic device has video playback, music playback, etc.), device identification information (such as the MAC address, IP address, SN, etc. of the electronic device), historical usage record information (such as the historical usage time and historical usage frequency of the user using the electronic device), device location information (such as the electronic device is placed in the kitchen, living room or bedroom, etc.), device status information (the remaining power of the electronic device, etc.), and service status suitability information (such as whether the electronic device is suitable for performing music playback or displaying push messages when the electronic device is performing video playback).

[0120] Secondly, the electronic device performs registration with the main control device, which can be understood as the electronic device uploading (or periodically uploading, uploading in response to a request from a registered device, etc.) its own device identification information, its own service function information, its own device status information, its own service status suitability information, and its own device wake-up information input by the user to the main control device, which then stores or manages the information, thereby completing the registration of the electronic device.

[0121] Again, the electronic devices in the registered device group may be equipped with an intelligent voice assistant or an intelligent gesture assistant, so that the electronic device can realize the voice control function or the gesture control function. At the same time, each electronic device in the registered device group is in the on or powered-on state, and the intelligent voice assistant or intelligent gesture assistant in each electronic device may be in a dormant state to save power. Since the intelligent voice assistant or intelligent gesture assistant may be in a dormant state, before the user needs to use the voice control function or gesture control function of the electronic device, a wake-up operation (such as voice wake-up or gesture wake-up) is required for the intelligent voice assistant or intelligent gesture assistant. Among them, the device wake-up information for waking up the intelligent voice assistant can be called a wake-up word (such as "Xiaobu Xiaobu"), and the device wake-up information for waking up the intelligent gesture assistant can be called a wake-up gesture (such as an "OK" gesture). At the same time, the wake-up word or the wake-up gesture can be pre-registered or stored in the electronic device.

[0122] Next, waking up the smart voice assistant may refer to the electronic device responding to the wake-up word input by the user to start the smart voice assistant, and waking up the smart gesture assistant may refer to the electronic device responding to the wake-up gesture input by the user to start the smart gesture assistant. In addition, the above-mentioned voice control function may refer to that after the smart voice assistant of the electronic device is started, the user inputs voice control information (such as a voice of "playing the news broadcast on TV") to the electronic device, thereby triggering the electronic device to automatically execute the service function identified by the voice control information. The above-mentioned gesture control function may refer to that after the smart gesture assistant of the electronic device is started, the user inputs gesture operation information to the electronic device, thereby triggering the electronic device to automatically execute the service function identified by the gesture control information.

[0123] Finally, each electronic device in the registered device group can be associated with the user identification information of the first user. The user identification information may include at least one of physiological feature information (such as hand shape, fingerprint, voiceprint, iris, face, etc.) and user account information (such as user account number, etc.). For example, the first user has logged in to each electronic device in the registered device using the same user account; the first user has used voice / face / fingerprint / gesture unlocking or verification functions to open each electronic device in the registered device, etc.

[0124] Specifically, the main control device can determine the target device that the first user wants to control from the registered device group based on the first information, so as to control the target device to perform the service function required by the first user. If the first information includes voice control information, and the voice control information is a voice signal, the main control device can convert the voice signal into computer-readable text through technologies such as ASR, and then analyze the computer-readable text through technologies such as natural language processing (NLP) and natural language understanding (NLU), thereby obtaining the service function identified by the voice signal. For example, when the voice signal is "play news broadcast", the service function identified by the voice signal is video playback, etc.

[0125] S630: The main control device determines electronic devices capable of executing the target service function identified by the first information from the registered device group to obtain a candidate device group.

[0126] Each electronic device in the candidate device group has a function of executing the target service.

[0127] It should be noted that the embodiment of the present application first screens out a candidate device group capable of executing the target service function from the registered device group (i.e., the initial screening), and then screens out the target device that ultimately executes the target service function from the candidate device group (i.e., the second screening). Since the master control device only needs to preliminarily screen out electronic devices capable of executing the target service function in the initial screening, and the master control device needs to make a joint decision based on the user decision parameters and status information set of the first user in the second screening, which requires a relatively large amount of calculation, the initial screening can reduce the amount of calculation in the second screening, which is beneficial to improving the efficiency of collaborative control between devices, ensuring that the target device decided is more in line with the user's intention, and thus improving the accuracy of device decisions to provide users with more accurate personalized decisions and better meet the personalized needs of users.

[0128] In one possible example, determining electronic devices capable of executing the target service function identified by the first information from the registered device group to obtain a candidate device group may include the following steps: identifying the first information to obtain the target service function; obtaining the device registration information of each electronic device in the registered device group; and determining electronic devices capable of executing the target service function from the registered device group based on the device registration information to obtain a candidate device group.

[0129] It should be noted that the master device can obtain the device registration information of the electronic device during the registration process of the electronic device for storage or management. The electronic device can upload the device registration information to the master device to complete the registration when establishing an initial communication connection with the master device, or can upload the device registration information and the device wake-up information to the master device together to complete the registration when obtaining the device wake-up information input by the user, and there is no specific limitation on this.

[0130] Specifically, the device registration information may include at least one of distance information, service function information, device identification information, historical usage record information, device location information, device status information, and service status suitability information.

[0131] Furthermore, the distance information may be used to indicate the distance between the user and the electronic device.

[0132] It should be noted that, first, the distance between the user and the electronic device can be determined by the energy intensity of the voice signal (such as device wake-up information, voice control information) detected by the microphone. Among them, the electronic device or the main control device in the registered device group can pre-store a voice intensity detection algorithm, and detect the energy intensity of the voice signal according to the voice intensity detection algorithm to determine the distance between the user and the electronic device.

[0133] Secondly, the distance between the user and the electronic device can be determined by the image size in the video frame, image frame sequence or image frame detected by the camera module. Among them, the electronic device in the registered device group or the main control device can calculate the distance between the user and the electronic device through the pinhole imaging principle and the image size.

[0134] Finally, the device registration information may not include distance information, that is, the electronic device to be registered does not need to calculate the distance between the user and itself, but uploads the detected voice signal (or detected image frame) to the main control device, and then the main control device calculates the distance between the user and the electronic device to be registered based on the energy intensity of the voice signal (or the pinhole imaging principle) for storage and management.

[0135] Furthermore, the service function information can be used to indicate the type of predetermined service function possessed by the electronic device. The predetermined service function possessed by the electronic device refers to at least one predetermined action operation that the electronic device can perform by calling an internal application or instruction. For example, when the electronic device is a smart phone, the mobile phone can perform video playback, music playback, phone calls, email sending and receiving, weather information reception, multimedia information display, game screen control, photo shooting, screen projection, etc. by calling an internal application or instruction.

[0136] It should be noted that the master device can determine electronic devices that match the target service function identified by the first information from the registered device group through the service function information, thereby forming the matching electronic devices into a candidate device group. In addition, during the registration process with the master device, the electronic device can upload its own service function information to the master device for storage or management, so as to complete the registration.

[0137] Furthermore, the device identification information may be used to indicate the identification information of the electronic device, for example, the MAC address, IP address, SN, etc. of the electronic device.

[0138] It should be noted that the master device can distinguish the electronic devices in the registered device group through the device identification information; can issue control commands to the corresponding electronic devices (such as the target device) through the device identification information; can query the user identification information associated with the device through the device identification information, etc. In addition, the electronic device can upload its own device identification information to the master device for storage or management during the registration process with the master device, so as to complete the registration.

[0139] Furthermore, the historical usage record information may be used to indicate the historical records of the user's use of the electronic device, wherein the historical usage record information may include at least one of historical usage time information and historical usage frequency information.

[0140] It should be noted that, since the electronic devices in the registered device group can be associated with the user identification information of the first user, that is, the device identification information is associated with the user identification information, the master device can store the historical usage record information of each electronic device in the registered device group used by the first user, so that the master device can determine which electronic devices in the registered device group the first user has recently used (determined by the historical usage time information) or which electronic devices in the registered device group the first user has used most frequently (determined by the historical usage frequency information) through the historical usage time information. In addition, the electronic device can upload its own historical usage record information to the master device for storage during the registration process with the master device, so as to complete the registration.

[0141] Furthermore, the device location information can be used to indicate the location where the electronic device is currently placed, for example, the electronic device is placed in the kitchen, living room or bedroom.

[0142] It should be noted that the master control device can determine the candidate device group (or the target device that the first user wants to control) by the device location information of the electronic devices in the registered device group. For example, when the registered device group includes a smart TV placed in the living room and a smart TV placed in the bedroom, the master control device can exclude the smart TV placed in the bedroom from the candidate device group (or target device); when the registered device group includes a smart TV placed in the living room and a smart refrigerator placed in the kitchen, the master control device can exclude the smart refrigerator placed in the kitchen from the candidate device group (or target device). In addition, the electronic device can upload its own device location information to the master control device for storage or management during the registration process with the master control device to complete the registration.

[0143] Furthermore, the device status information may be used to indicate the current device status of the electronic device, for example, the current remaining power of the electronic device.

[0144] It should be noted that the master device can determine the candidate device group (or the target device that the first user wants to control) through the device status information of the electronic devices in the registered device group. For example, when the registered device group includes mobile phones with low battery and mobile phones with sufficient battery, the master device can exclude the mobile phones with low battery from the candidate device group (or target device). In addition, the electronic device can upload its own device status information to the master device for storage or management during the registration process with the master device, so as to complete the registration.

[0145] Furthermore, the service status suitability information can be used to indicate whether the electronic device is suitable for executing other service functions when currently performing a certain service function.

[0146] It should be noted that the master device can determine the candidate device group (or the target device that the first user wants to control) by registering the service status suitability information of the electronic devices in the device group. For example, when the target service function identified by the first information is music playback, if the mobile phone is performing video playback, the mobile phone will not be suitable for performing music playback, so the mobile phone needs to be excluded from the candidate device group (or target device). In addition, the electronic device can upload its own service status suitability information to the master device for storage or management during the registration process with the master device to complete the registration.

[0147] The following embodiments of the present application will explain how to identify the first information to obtain the target service function.

[0148] In one possible example, identifying the first information to obtain the target service function may include the following steps: if the first information includes voice control information, determining the target service function based on the word vector matrix identified by the voice control information and a pre-trained service function analysis model.

[0149] Among them, the voice control information can be a voice signal, voice data, etc.

[0150] Specifically, determining the target service function based on the word vector matrix identified by the voice control information and the pre-trained service function analysis model can include the following steps: performing a text conversion operation on the voice control information to obtain text data information; performing Chinese word segmentation processing on each character in the text data information to obtain a target word group, and all the characters in the target word group are used to form a sentence in the text data information; inputting all the characters in the target word group into a pre-trained word vector model to obtain a word vector matrix; and inputting the word vector matrix into the pre-trained service function analysis model to obtain the target service function.

[0151] It should be noted that, first of all, the word vector model can include one of the following: one-hot model, bag of words (BOW) model, continuous bag-of-words (CBOW) model, continuous skip grammar (Skip-Gram) model.

[0152] Secondly, the service function analysis model includes an input layer, two layers of long short-term memory (LSTM) neural layers and a Softmax function layer. The word vector matrix is ​​input into the two layers of LSTM neural layers through the input layer; each layer of the two layers of LSTM neural layers has a total of multiple (such as 32, 64, 128, 256 or 512) neurons, and the multiple neurons are fully connected to achieve positive and negative mapping; the Softmax function layer is used to classify service functions, such as Figure 7 It should be noted that, since the Softmax function layer in the service function analysis model can output at least one probability value, the embodiment of the present application uses the probability value with the maximum value among the at least one probability value as the target service function.

[0153] Finally, each LSTM neural layer in the two LSTM neural layers in the embodiment of the present application introduces a dropout strategy, that is, randomly discarding parameters of a preset threshold (such as 20%) to prevent overfitting and increase the calculation rate. At the same time, the window batch size (batch) is set between [10,16], and the root mean square propagation (RMSProp) optimization algorithm is adopted, and its parameter learning rate is set to 0.001 and the fuzzy factor is 1e -8 , thereby ensuring that the output results of the service function analysis model are more accurate.

[0154] In one possible example, identifying the first information to obtain the target service function may include the following steps: if the first information includes gesture control information, determining the target service capability based on the target gesture type identified in the gesture control information and a preset mapping relationship, and the preset mapping relationship is used to indicate the mapping relationship between the target gesture type and the target service capability.

[0155] The target gesture type can be determined by the coordinate information of the key points of the hand.

[0156] The gesture control information may be a video frame, an image frame sequence, an image frame, or a frame image.

[0157] It should be noted that gesture type refers to the types of various gestures made by the user with his hands (one hand or two hands). For example, the "fist" gesture, the "waving hand" gesture, the "OK" gesture, the "heart" gesture, the "victory" gesture, the "number" gesture, the "prayer" gesture, the "like" gesture, the "middle finger" gesture, etc. Therefore, the embodiment of the present application can find the mapping relationship between the target gesture type and the target service capability through a preset mapping relationship, so as to obtain the target service type by identifying the gesture control information. For example, the "fist" gesture corresponds to "video playback" and the like.

[0158] In addition, the key points of the hand in the embodiment of the present application refer to the key positions of the hand skeleton, wherein the key points of the hand may include the fingertips of each finger (i.e., thumb, index finger, middle finger, ring finger and little finger), the joints of the phalanges of each finger, etc.

[0159] For example, see Figure 8, the hand 800 may include carpal bones, five metacarpal bones and five phalanges. The five metacarpal bones are respectively connected to the five phalanges, and the five phalanges include the thumb bone, the index finger bone, the middle phalange, the ring finger bone and the little finger bone. The thumb bone is divided into two sections, while the index finger bone, the middle phalange, the ring finger bone and the little finger bone are each divided into three sections. In addition, along the direction toward the wrist, the metacarpal bone connected to the thumb bone can be divided into a first metacarpal bone and a second metacarpal bone. In summary, the hand can include the following 21 key points: the connection point k0 between the carpal bones and each metacarpal bone; the connection point k1 between the first metacarpal bone and the second metacarpal bone; the connection point k2 between the second segment of the thumb bone and the first metacarpal bone; the connection point k3 between the second segment of the thumb bone and the first segment of the thumb bone; the tip point k4 of the thumb bone; the connection point k5 between the third segment of the index finger bone and the metacarpal bone connected to the same index finger bone; the connection point k6 between the second segment of the index finger bone and the third segment of the index finger bone; the connection point k7 between the first segment of the index finger bone and the second segment of the index finger bone; the tip point k8 of the index finger bone; the connection point k9 between the third segment of the middle phalanx and the metacarpal bone connected to the same middle phalanx; the second segment of the middle phalanx and the third segment of the middle phalanx The connection point k10 of the first segment of the middle phalanx and the second segment of the middle phalanx; k11; the tip point of the middle phalanx k12; the connection point k13 of the third segment of the ring phalanx and the metacarpal bone connected to the ring phalanx; k14; the connection point k15; the connection point k16 of the ring phalanx; the connection point k17 of the third segment of the little finger bone and the metacarpal bone connected to the little finger bone; k18; the connection point k19; the connection point k20 of the little finger bone.

[0160] Specifically, determining the target service capability based on the target gesture type identified in the gesture control information and the preset mapping relationship can include the following steps: determining a target area in the gesture control information, the target area being used to frame the boundary of the imaging of the first user's hand on the gesture control information; determining the coordinate information of the hand key points within the target area; determining the target gesture type based on the coordinate information of the hand key points; determining the target service capability based on the target gesture type and the preset mapping relationship.

[0161] Furthermore, determining the target area on the gesture control information may include the following steps: determining the target area on the gesture control information by using a target detection algorithm.

[0162] It should be noted that target detection is defined as identifying which objects are in an image and their locations (coordinate positions). The objects to be identified can be set by the user, and the coordinate positions of the objects are represented by polar coordinates (xmin, ymin, xmax, ymax) or center point coordinates (x_center, y_center, w, h). Therefore, the target area can be regarded as a bounding box in the target detection algorithm, which can be a bounding box after bounding-box regression or a ground truth bounding box.

[0163] In addition, the target detection algorithm may include a one-stage detection algorithm and a two-stage detection algorithm. Among them, the one-stage detection algorithm may include a single-shot multi-box detector (SSD), a you only look once (YOLO) algorithm, etc.; the two-stage detection algorithm may include a region-convolutional neural network (R-CNN), a fast R-CNN, a faster R-CNN, a region-based fully convolutional network (R-FCN), etc.

[0164] Furthermore, determining the coordinate information of the hand key points in the target area may include the following steps: determining the hand key points in the target area through a hand key point detection model; determining the coordinate information of the hand key points in a first coordinate system, wherein the first coordinate system is constructed by a two-dimensional plane where the gesture control information is located.

[0165] It should be noted that for the gesture key point detection model, first, the embodiment of the present application uses a small amount of annotated data set containing hand key points to train a convolutional pose machines (CPMs) model; secondly, multiple groups of hand images are obtained by shooting with a camera, and the multiple groups of hand images are input into the trained CPMs model to obtain preliminary hand key points; thirdly, the preliminary hand key points are triangulated according to the camera posture to obtain the 3D coordinate position of the hand key points, and then the 3D coordinate position is projected to each 2D image of different perspectives; finally, the 2D image and the preliminary hand key points are used to train the CPMs model again, and after several iterations, a more accurate hand key point detection model is obtained.

[0166] For example, see Fig. 9 .exist Fig. 9 In the figure, image frame 900 represents gesture control information. The two-dimensional plane where image frame 900 is located is a two-dimensional coordinate system (i.e., a first coordinate system) constructed by the xy axis, and the coordinate origin of the two-dimensional coordinate system is the center point of image frame 900. Secondly, the various hand key points in the target area 910 are determined by the above-mentioned gesture key point detection model. Finally, the coordinate information of each hand key point in the two-dimensional coordinate system is calculated, and the gesture type in image frame 900 is determined according to the coordinate information.

[0167] S640: The main control device obtains a user decision parameter of the first user and first state information of the first candidate device.

[0168] The first candidate device may be a candidate device in the candidate device group, and the first status information may be a status information in the status information set.

[0169] The user decision parameter may be used to indicate the historical decision made by the first user, the usage habits of the first user, or the decision habits of the first user, etc. For example, the user decision parameter indicates that the user tends to select electronic devices that are close and have a high historical usage frequency.

[0170] Specifically, the user decision parameter may include a preset initial value. It should be noted that if the master control device does not pre-store the user decision parameter of the first user, that is, the first user is a new user, the master control device may select the preset initial value according to the usage habits of multiple users based on big data statistics. The master control device may update and optimize the preset initial value based on the first user's subsequent response to the request information of the master control device.

[0171] Specifically, the user decision parameter may include a historical decision parameter, which may be used to indicate the electronic device to be controlled selected by the first user in the historical response to the request information of the master device (i.e., the historical decision made by the first user, the usage habits of the first user, the decision habits of the first user, etc.), and the request information may be used to request the first user to select the electronic device to be controlled. Alternatively, the historical decision parameter may be used to indicate the electronic device to be controlled adjusted by the master device in the historical response to the device adjustment information of the first user (i.e., the historical decision made by the first user, the usage habits of the first user, the decision habits of the first user, etc.), and the device adjustment information may be used to request the master device to adjust the electronic device that performs the target service function.

[0172] The first state information may be used to indicate a multidimensional state feature between the first user and the first candidate device or a multidimensional state feature of the first candidate device. The multidimensional state feature between the first user and the first candidate device may include at least one of geographical distance, historical usage time, and historical usage frequency. The multidimensional state feature of the first candidate device may include at least one of device state, service state suitability, and device location.

[0173] It should be noted that, first of all, in the information processing scheme, the embodiment of the present application can adopt a single device selection strategy. For example, the main control device selects the target device only according to the service function provided by the electronic device, selects the electronic device closest to the user as the target device, selects the electronic device with the highest frequency of use by the user as the target device, or selects the electronic device most recently used by the user. However, the use of a single device selection strategy may not be able to adapt to the complex situation in the current environment. Therefore, the embodiment of the present application also needs to consider a multi-dimensional device selection strategy, that is, the multi-dimensional state characteristics between the user and the candidate devices and / or the multi-dimensional state characteristics of each candidate device, so as to make decisions through the multi-dimensional state characteristics between the user and each electronic device and / or the multi-dimensional state characteristics of the electronic device, avoid the problem of overly simple single-dimensional decision-making, and better adapt to complex scenarios in the real environment.

[0174] For example, see Fig.10The candidate device group includes a smart TV 1011, a smart phone 1012, a laptop computer 1013, and a smart phone 1014. The distance between the user 1020 and the smart TV 1011 is 0.5 m, the distance between the user 1020 and the smart phone 1012 is 0.6 m, the distance between the user 1020 and the laptop computer 1013 is 1.2 m, the distance between the user 1020 and the smart phone 1014 is 0.55 m, the historical usage frequency of the user 1020 using the smart TV 1011 is 20 times, the historical usage frequency of the user 1020 using the smart phone 1012 is 60 times, the historical usage frequency of the user 1020 using the laptop computer 1013 is 70 times, and the historical usage frequency of the user 1020 using the smart phone 1014 is 35 times. When the historical decision (or decision habit, usage habit) of the user 1020 is to select an electronic device that is close and has a high historical usage frequency, according to the historical decision of the user 1020, the smart phone 1012 or the smart phone 1014 may be used as the target device for executing the service function required by the user 1020. However, according to a single device selection strategy, the smart TV 1011 or the laptop may be used as the target device for executing the service function required by the user 1020, resulting in the inability to accurately identify the user's intention.

[0175] Secondly, the device selection strategy implemented for different users may be different. For example, some users are accustomed to using the device that is closest to them. Fig.10 The smart TV 1011 in the example is used as the target device, while some users are used to using the most commonly used device. Fig.10 The laptop computer 1013 in is taken as the target device. Therefore, in order to consider the usage habits of different users, avoid using the same device selection strategy for different users, and avoid using a single device selection strategy, the embodiment of the present application not only considers the user decision parameter of the first user (used to indicate the historical decision made by the first user, the usage habits of the first user, or the decision habits of the first user, etc.) as a factor of the device selection strategy, but also considers the multi-dimensional state characteristics between the user and the candidate device or the multi-dimensional state characteristics of the candidate device, so as to realize personalized device decision according to the user's usage habits (indicated by the user decision parameter) and multi-dimensional characteristics (indicated by the state information), ensure that the selected device is more in line with the user's needs or intentions, and thus help to provide a better user experience, ensure the user's personalized decision, and meet the user's personalized needs.

[0176] Finally, the main control device can obtain, store or manage the user decision parameters of the first user. This is because: First: if the main control device does not pre-store the user decision parameters of the first user, that is, the first user is a new user, the main control device can select the preset initial value according to the usage habits of multiple users based on big data statistics; Second: if the main control device pre-stores the user decision parameters of the first user, that is, the first user is not a new user, then when there are multiple electronic devices in the candidate device group determined by the main control device, the main control device can generate a request information for requesting the first user to select the electronic device to be controlled from the candidate device group, so that the main control device can obtain the historical decision or usage habits (indicated by the user decision parameters) made by the first user in response to the request information for updating. Among them, the main control device can associate the user identification information of the first user and the identification information of the electronic device to be controlled to obtain the user decision parameters.

[0177] Exemplarily, take the controlled device as a smart phone and the main control device as a server as an example. First, the user inputs the voice control information of "playing the news broadcast" through the smart phone, and uploads the voice control information to the server through the smart phone. Secondly, the server determines the candidate device group of the target service function (i.e., broadcasting video) identified by the voice control information from the registered device group, and the candidate device group includes smart TVs, laptops, and tablets. Again, the server sends a request message to the smart phone, "Which device do you want to play the news broadcast: 1. Smart TV; 2. Laptop; 3. Tablet?". Immediately afterwards, the user selects "1. Smart TV" through the smart phone to respond to the request information, and feeds back the selection result to the server. At this time, the smart TV serves as the user's electronic device to be controlled. Finally, the server sends a control command for executing "playing the news broadcast" to the smart TV according to the selection result, and the server associates the user identification information of the user and the identification information of the smart TV to obtain the user's user decision parameters.

[0178] Specifically, the first status information may include at least one of the distance information between the first user and the first candidate device, the historical usage time information of the first user using the first candidate device, the historical usage frequency information of the first user using the first candidate device, the device status information of the first candidate device, and the service status suitability information of the first candidate device.

[0179] It should be noted that the main control device can obtain the state information set through stored device registration information, user identification information of the first user, device identification information of the electronic device, and the like.

[0180] S650: The master control device determines a target device from the candidate device group according to the user decision parameter and the state information set.

[0181] It should be noted that in the scenario of collaborative control of multiple devices, the intelligent voice assistant or intelligent gesture assistant may not be able to determine the electronic device that the user wants to control based only on the user's voice control information or gesture control information. Fig.10 In the example, when a user issues a "play music" command, the collaborative control system between devices may not be able to determine which electronic device should be selected to provide the required service functions for the user. In this case, the embodiment of the present application considers that the intelligent voice assistant or voice gesture assistant should infer the user's required target device through the user's usage habits, the multidimensional state characteristics between the user and each candidate device, and / or the multidimensional state characteristics of each candidate device. Therefore, the embodiment of the present application makes personalized device decisions based on the user's usage habits (indicated by user decision parameters) and multidimensional characteristics (indicated by status information) to ensure that the selected device is more in line with the user's needs or intentions, which is conducive to providing a better user experience, ensuring the user's personalized decision, and meeting the user's personalized needs.

[0182] In addition, the embodiment of the present application first selects a candidate device group capable of executing the target service function from the registered device group (i.e., the first screening), and then selects the target device that ultimately executes the target service function from the candidate device group (i.e., the second screening). Since the master control device only needs to preliminarily select electronic devices capable of executing the target service function in the first screening, and the master control device needs to make a joint decision based on the user decision parameters and status information set of the first user in the second screening, which requires a relatively large amount of calculation, a second screening (i.e., the second screening) is performed on the basis of the first screening, which is conducive to improving the efficiency of collaborative control between devices, ensuring that the target device determined is more in line with the user's intention, and improving the accuracy of device decisions, thereby providing users with more accurate personalized decisions and more personalized needs of users.

[0183] In one possible example, determining a target device from a candidate device group based on a user decision parameter and a state information set may include the following steps: determining a first parameter value for a first candidate device based on the user decision parameter and the state information set, the first parameter value being used to indicate the probability of the first candidate device being the one that executes the target service function, the first parameter value being a parameter value in a parameter value set; and determining the target device from the candidate device group based on the parameter value set and a preset threshold.

[0184] It should be noted that the embodiment of the present application determines the probability of each candidate device in the candidate device group as executing the target service function based on the user decision parameters and status information of the first user to obtain a parameter value set, and then determines the target device from the candidate device group through the comparison result between the parameter value set and the preset threshold value, so as to realize personalized device decision (i.e., determine the target device) according to the user's usage habits (indicated by the user decision parameters) and multi-dimensional characteristics (indicated by the multi-dimensional status information) through the probability size of each device, thereby ensuring that the selected device is more in line with the user's needs or intentions, which is conducive to providing a better user experience, ensuring the user's personalized decision, and meeting the user's personalized needs.

[0185] Specifically, determining the first parameter value for the first candidate device based on the user decision parameters and the state information set may include the following steps: inputting the user decision parameters and each state information in the state information set into a pre-trained device decision model to obtain the probability size of each candidate device in the candidate device group as executing the target service function, and forming a parameter value set with the probability size of each candidate device as executing the target service function, wherein the parameter value set includes the first parameter value.

[0186] It should be noted that the pre-trained device decision model can be a pre-trained neural network module. It can be seen that the pre-trained device decision module can ensure that the output probability size is more accurate. The first parameter value is illustrated below.

[0187] Exemplarily, the first parameter value is calculated as follows:

[0188] P i =f(h i,1 ,h i,2 ,...,h i,N ,θ),i∈{1,2,...,M};

[0189] Among them, P i represents the first parameter value; M represents the total number of candidate devices in the candidate device group, that is, the first candidate device is the i-th candidate device in the candidate device group; N represents the total number of single-dimensional information in the state information (i.e., the first state information) for the i-th candidate device; h i,j ,j∈{1,2,...,N} represents the jth single-dimensional information in the status information of the i-th candidate device, such as h i,j represents the historical frequency of use of the i-th candidate device by the first user, etc.; θ represents the user decision parameter of the first user; f(·) represents the device decision function, and in engineering implementation, f(·) can be a neural network model.

[0190] Specifically, determining the target device from the candidate device group according to the parameter value set and the preset threshold may include: determining the target device from the candidate device group according to a comparison result between a target parameter value in the parameter value set and the preset threshold, wherein the target parameter value is the maximum value in the parameter value set.

[0191] In one possible example, determining a target device from a candidate device group based on a parameter value set and a preset threshold may include the following steps: if a target parameter value in the parameter value set is greater than or equal to the preset threshold, the candidate device corresponding to the target parameter value is taken as the target device, and the target parameter value is the maximum value in the parameter value set.

[0192] It should be noted that the embodiment of the present application compares the maximum value in the parameter value set with the preset threshold value. If the maximum value is greater than or equal to the preset threshold value, the master control device uses the candidate device corresponding to the maximum value as the target device. In other words, if the probability of a candidate device in the candidate device group executing the target service function is the maximum value, and the maximum value is greater than or equal to the preset threshold value, the candidate device is used as the target device, and a control instruction to execute the target service function is issued to the candidate device, thereby ensuring that the selected device is more in line with the user's needs or intentions through the judgment result of the maximum value and the preset threshold value, which is conducive to providing a better user experience, ensuring the user's personalized decision, and meeting the user's personalized needs.

[0193] In one possible example, determining a target device from a candidate device group based on a parameter value set and a preset threshold may include the following steps: if the target parameter value is less than the preset threshold, generating a request message for requesting a first user to select a candidate device to be controlled from the candidate device group, and sending the request message to the controlled device; obtaining response information from the controlled device for the request information; and using the candidate device to be controlled indicated by the response information as the target device.

[0194] It should be noted that the embodiment of the present application compares the maximum value in the parameter value set with the preset threshold value. If the maximum value is less than the preset threshold value, the master device needs to send a request message to the controlled device to request the first user to select a candidate device to be controlled from the candidate device group to perform the target service function. Then, the first user inputs a response message in response to the request message through the controlled device, and the controlled device sends the response message to the master device, and the response message is used to indicate that the device to be controlled in the candidate device group is the target device. Finally, the master device receives the response message and sends a control command for indicating the target service function to the candidate device to be controlled indicated by the response message, thereby realizing the collaborative interaction process between the collaborative control system between devices and the user through the sending of the request information, ensuring that the selected device is more in line with the user's needs or intentions, and thus facilitating providing a better user experience, ensuring the user's personalized decision, and meeting the user's personalized needs.

[0195] Specifically, after obtaining response information from the controlled device to the request information, the method may further include the following steps: updating the user decision parameter according to the response information.

[0196] It should be noted that, in combination with the above description, the user decision parameter may include a historical decision parameter, and the historical decision parameter may be used to indicate the electronic device to be controlled selected by the first user in historical response to the request information of the master device. Therefore, the embodiment of the present application considers that the master device can update and optimize the user's usage habits (indicated by the user decision parameter) in real time according to the response information fed back by the user, thereby providing the user with more accurate personalized decisions and more satisfying the user's personalized needs, thereby ensuring that the communication process between the intelligent voice assistant or the intelligent gesture assistant and the user is smoother and more intelligent.

[0197] In a possible example, determining a target device from a candidate device group according to a parameter value set and a preset threshold may include the following steps: taking at least one candidate device corresponding to a parameter value in the parameter value set that is greater than or equal to the preset threshold as the target device.

[0198] It should be noted that, unlike the above example, the embodiment of the present application considers that the target device may include at least one candidate device. At this time, the master control device can simultaneously issue a control instruction for executing the target service function to the at least one candidate device, and then the at least one candidate device simultaneously executes the target service function, instead of being limited to using a candidate device with the maximum probability as the target device, thereby achieving simultaneous control of multiple devices to meet different user needs and scenarios.

[0199] S660: The master control device sends second information to the target device.

[0200] The second information may be used to control the target device to execute the target service function.

[0201] It should be noted that the second information may be a control command that can be read or executed by the target device.

[0202] In a possible example, after sending the second information to the target device, the method may further include the following steps: receiving device adjustment information from the target device, where the device adjustment information is used to request the main control device to adjust the electronic device that executes the target service function.

[0203] It should be noted that after the master device sends a control command for executing the target service function to the target device, if the first user needs to adjust the electronic device that executes the target service function, that is, the first user needs to adjust the target device, the first user can send device adjustment information to the master device through the target device. It can be seen that the electronic devices (such as each electronic device in the registered device group, each candidate device in the candidate device group, the controlled device, the target device, etc.) and the master device of the embodiment of the present application have or provide device adjustment functions, thereby providing a variety of device decision strategies through device adjustment, thereby providing a better user experience, ensuring the user's personalized decision, and meeting the user's personalized needs.

[0204] Specifically, after receiving the device adjustment information from the target device, the method further includes: updating the user decision parameter according to the device adjustment information.

[0205] It should be noted that, in combination with the above description, it can be seen that the embodiment of the present application considers that the main control device can update and optimize the user's usage habits (indicated by user decision parameters) in real time according to the device adjustment information fed back by the user, thereby providing the user with more accurate personalized decisions and better meeting the user's personalized needs, thereby ensuring that the communication process between the intelligent voice assistant or intelligent gesture assistant and the user is smoother and more intelligent.

[0206] Exemplarily, the update calculation of the user decision parameter is as follows:

[0207]

[0208] Among them, θ represents the user decision parameter updated according to the device adjustment information; y k represents the kth electronic device (indicated by the device adjustment information) in the candidate device group for performing the target service function after the device adjustment, and y k =1; f(h i,1 ,h i,2 ,...,h i,N ,θ) represents the prediction result determined by the master device according to θ and the status information of the kth electronic device; L(y k ,f(hk,1 ,h k,2 ,...,h k,L ,θ)) represents y k With f(h i,1 ,h i,2 ,...,h i,N ,θ); λ represents the preset hyperparameter defined by the master device; θ old Indicates the user decision parameters before the device adjustment information is updated; Used to limit θ to avoid θ and θ old The difference is too large, resulting in unstable prediction results.

[0209] In combination with the above description, the embodiment of the present application further provides an example to illustrate the process of the information processing method.

[0210] For example, see Fig.11 The information processing methods include:

[0211] S1101: A user inputs a device control command to a controlled device.

[0212] Among them, the device control command can be regarded as the above-mentioned first information.

[0213] Among them, the controlled device sends the device control command to the main control device.

[0214] S1102: The main control device determines whether the target device can be determined according to the device control command.

[0215] For example, if the device control command is "play music on a smartphone", the main control device can directly identify the target device as a "smartphone".

[0216] Among them, the main control device can identify the device control command to obtain the target service function.

[0217] If S1102 cannot determine the target device, then S1103 is executed; if S1102 can determine the target device, then S1111 is executed.

[0218] S1103: The main control device determines electronic devices capable of executing the target service function according to the device control command to obtain a candidate device group.

[0219] The main control device may determine the candidate device group according to the service capability information of the electronic device.

[0220] S1104: The main control device obtains a user decision parameter and a state information set of the user.

[0221] The state information set includes the state information of each candidate device in the candidate device group.

[0222] The state information includes at least one single-dimension state feature, which may be a historical usage time feature, a distance feature, or a historical usage frequency feature.

[0223] S1105. The device decision module of the master device determines the probability of each candidate device in the candidate device set to execute the target service function according to the user decision parameter and the state information set to obtain a parameter value set.

[0224] Among them, the equipment decision module can be a pre-selected and trained equipment decision model.

[0225] S1106: The master control device determines whether there is a parameter value in the parameter value set that is higher than a preset threshold.

[0226] The master control device may make a judgment based on the maximum value in the parameter value set.

[0227] If there is a parameter value higher than the preset threshold, execute S1107; if there is no parameter value higher than the preset threshold, execute S1108.

[0228] S1107: The master control device takes the candidate device corresponding to the parameter value higher than the preset threshold as the target device.

[0229] S1108. The main control device sends a request message to the user.

[0230] The request information is used to request the user to select a candidate device to be controlled from the candidate device group.

[0231] The user inputs response information in response to the request information through the controlled device, and the controlled device sends the response information to the main control device.

[0232] After receiving the response information, the main control device executes S1109 and S1110.

[0233] S1109: The master control device determines the target device according to the response information to the request information.

[0234] S1110. The main control device updates the user decision parameters according to the response information.

[0235] Among them, by updating the user decision parameters, it is ensured that the decision of the device decision module is more in line with the user's usage habits.

[0236] S1111. The master control device controls the target device to execute a target service function.

[0237] The main control device may send a control command to the target device, and the control command is used to control the target device to execute the target service function.

[0238] S1112: Whether the user needs to adjust the electronic device that executes the target service function.

[0239] If adjustment is required, the user sends device adjustment information to the main control device through the target device, and the device adjustment information is used to request the main control device to adjust the electronic device that executes the target service function, and execute S1113.

[0240] S1113. The main control device updates the user decision parameters according to the device adjustment information.

[0241] Among them, by updating the user decision parameters, it is ensured that the decision of the device decision module is more in line with the user's usage habits.

[0242] It should be noted that the description of each embodiment in this application has its own emphasis. Fig.11 For parts not described in detail in the embodiments, please refer to Figure 6 The relevant description in the embodiments will not be repeated in detail.

[0243] It can be seen that in the embodiment of the present application, first, the controlled device obtains the first information of the first user and sends the first information to the main control device; secondly, the main control device receives the first information from the controlled device, and determines, from the registered device group, an electronic device that has the target service function identified by the first information to obtain a candidate device group; thirdly, the main control device obtains the user decision parameters of the first user and the first status information of the first candidate device; finally, the main control device determines the target device from the candidate device group according to the user decision parameters and the status information set, and sends the second information to the target device. Since the embodiment of the present application first screens out a candidate device group capable of executing the target service function from the registered device group (i.e., the initial screening), and then screens out the target device that ultimately executes the target service function from the candidate device group (i.e., the second screening), in the initial screening, the main control device only needs to simply and preliminarily screen out the electronic devices capable of executing the target service function, while in the second screening, the main control device needs to make a joint decision based on the user decision parameters and state information set of the first user, which requires a relatively large amount of calculation. Therefore, by performing a second screening (i.e., secondary screening) on ​​the basis of the initial screening, it is beneficial to improve the efficiency of collaborative control between devices, ensure that the target device decided is more in line with the user's intention, and improve the accuracy of device decision. In addition, the embodiment of the present application makes personalized device decisions based on the user decision parameters and state information set of the first user, ensuring that the target device decided is more in line with the user's needs or intentions, providing users with more accurate personalized decisions and more personalized needs of users.

[0244] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that in order to realize the above functions, the master control device or the controlled device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments provided in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0245] The embodiment of the present application can divide the functional units of the master device or the controlled device according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division, and there may be other division methods in actual implementation.

[0246] In the case of an integrated unit, Fig.12 A block diagram of the functional units of an information processing device is provided. The information processing device 1200 is applied to a main control device, and specifically includes: a processing unit 1220 and a communication unit 1230. The processing unit 1220 is used to control and manage the actions of the main control device. For example, the processing unit 1220 is used to support the main control device to execute Figure 6 or Fig.11 The communication unit 1230 is used to support the communication between the master device and other devices. The information processing device 1200 may also include a storage unit 1210 for storing program codes and data of the information processing device 1200.

[0247] Among them, the processing unit 1220 can be a processor or a controller, such as a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, a transistor logic device, a hardware component, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present application. In addition, the processing unit 1220 can also be a combination that implements computing functions, such as a combination of one or more microprocessor combinations, a combination of DSPs and microprocessors. The communication unit 1230 can be a communication interface, a transceiver, a transceiver circuit, etc. The storage unit 1210 can be a memory. When the processing unit 1220 is a processor, the communication unit 1230 is a communication interface, and the storage unit 1210 is a memory, the information processing device 1200 involved in the embodiments of the present application can be Fig.14 The master device shown.

[0248] Specifically, the processing unit 1220 is used to execute any step executed by the main control device in the above method embodiment, and when performing data transmission such as sending, the communication unit 1230 can be selectively called to complete the corresponding operation.

[0249] The processing unit 1220 is used to: receive first information from a first user of a controlled device, where the controlled device is an electronic device determined in a registered device group in response to device wake-up information of the first user, and the registered device group is composed of electronic devices that are registered with the main control device; determine an electronic device capable of executing a target service function identified by the first information from the registered device group to obtain a candidate device group; obtain a user decision parameter of the first user and first status information of a first candidate device, where the first candidate device is a candidate device in the candidate device group, and the first status information is a status information in a status information set; determine a target device from the candidate device group according to the user decision parameter and the status information set; and send second information to the target device, where the second information is used to control the target device to execute the target service function.

[0250] It should be noted that the specific implementation of each operation performed by the information processing device 1200 can refer to the above Figure 6 or Fig.11 The corresponding description of the method embodiment shown will not be repeated here.

[0251] It can be seen that in the embodiment of the present application, since the embodiment of the present application first screens out the candidate device group with the target service function from the registered device group (i.e., the initial screening), and then screens out the target device that finally executes the target service function from the candidate device group (i.e., the second screening), in the initial screening, the main control device only needs to simply and preliminarily screen out the electronic devices with the target service function, and in the second screening, the main control device needs to make a joint decision based on the user decision parameters and state information set of the first user, which has a large amount of calculation. Therefore, by performing a second screening (i.e., secondary screening) on ​​the basis of the initial screening, it is beneficial to improve the efficiency of collaborative control between devices, ensure that the target device decided is more in line with the user's intention, and improve the accuracy of device decision. In addition, the embodiment of the present application makes personalized device decisions based on the user decision parameters and state information set of the first user, ensuring that the target device decided is more in line with the user's needs or intentions, providing users with more accurate personalized decisions and more personalized needs of users.

[0252] In one possible example, in terms of determining a target device from a candidate device group based on a user decision parameter and a state information set, the processing unit 1220 is specifically used to: determine a first parameter value for the first candidate device based on the user decision parameter and the state information set, the first parameter value being used to indicate the probability of the first candidate device being the one that executes the target service function, the first parameter value being a parameter value in a parameter value set; and determine the target device from the candidate device group based on the parameter value set and a preset threshold.

[0253] In one possible example, in terms of determining a target device from a candidate device group based on a parameter value set and a preset threshold, the processing unit 1220 is specifically used to: if the target parameter value in the parameter value set is greater than or equal to the preset threshold, then use the candidate device corresponding to the target parameter value as the target device, and the target parameter value is the maximum value in the parameter value set; or, if the target parameter value is less than the preset threshold, generate a request message for requesting the first user to select a candidate device to be controlled from the candidate device group, and send the request message to the controlled device; obtain response information from the controlled device for the request information; use the candidate device to be controlled indicated by the response information as the target device; or, use at least one candidate device corresponding to a parameter value in the parameter value set that is greater than or equal to the preset threshold as the target device.

[0254] In a possible example, after obtaining response information from the controlled device to the request information, the processing unit 1220 is further configured to: update the user decision parameter according to the response information.

[0255] In one possible example, in determining electronic devices capable of executing the target service function identified by the first information from a registered device group to obtain a candidate device group, the processing unit 1220 is specifically used to: identify the first information to obtain the target service function; obtain the device registration information of each electronic device in the registered device group; and determine electronic devices capable of executing the target service function from the registered device group based on the device registration information to obtain the candidate device group.

[0256] In one possible example, in terms of identifying first information to obtain a target service function, the processing unit 1220 is specifically used to: if the first information includes voice control information, then determine the target service function based on the word vector matrix identified by the voice control information and a pre-trained service function analysis model; or, if the first information includes gesture control information, then determine the target service capability based on the target gesture type identified in the gesture control information and a preset mapping relationship, and the preset mapping relationship is used to indicate the mapping relationship between the target gesture type and the target service capability.

[0257] In a possible example, after sending the second information to the target device, the processing unit 1220 is further used to: receive device adjustment information from the target device, where the device adjustment information is used to request the main control device to adjust the electronic device that executes the target service function.

[0258] In the case of an integrated unit, Fig.13 A functional unit block diagram of another information processing device is provided. The information processing device 1300 includes a processing unit 1320 and a communication unit 1330. The processing unit 1320 is used to control and manage the actions of the controlled device. For example, the processing unit 1320 is used to support the terminal to execute Figure 6 or Fig.11 The steps in and other processes of the technical solution described in this application. The communication unit 1330 is used to support communication between the controlled device and other devices. The information processing device 1300 may also include a storage unit 1310 for storing program codes executed by the information processing device 1300 and transmitted data.

[0259] Among them, the processing unit 1320 can be a processor or a controller, for example, it can be a CPU, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component or any combination thereof. It can implement or execute various exemplary logic boxes, modules and circuits described in combination with the contents disclosed in this application. The processing unit 1320 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, and so on. The communication unit 1330 can be a communication interface, a transceiver, a transceiver circuit, etc., and the storage unit 1310 can be a memory. When the processing unit 1320 is a processor, the communication unit 1330 is a communication interface, and the storage unit 1310 is a memory, the information processing device 1300 involved in the embodiment of the present application can be Fig.15 The controlled device shown.

[0260] In specific implementation, the processing unit 1320 is used to execute any step executed by the controlled device in the above method embodiment, and when performing data transmission such as sending, the communication unit 1330 can be selectively called to complete the corresponding operation.

[0261] The processing unit 1320 is used to: obtain first information of the first user, where the controlled device is an electronic device in the registered device group determined in response to the device wake-up information of the first user, and the registered device group is composed of electronic devices that register with the main control device; send first information to the main control device, and the first information is used by the main control device to determine the target device that executes the target service function identified by the first information from the registered device group.

[0262] It should be noted that the specific implementation of each operation performed by the information processing device 1300 can refer to the above Figure 6 or Fig.11 The corresponding description of the method embodiment shown will not be repeated here.

[0263] It can be seen that in the embodiment of the present application, by obtaining the first information of the first user and sending the first information to the master device, the first information is used by the master device to determine the target device that executes the target service function identified by the first information from the registered device group, thereby realizing the collaborative control operation between devices.

[0264] The following is a schematic diagram of the structure of a main control device provided by an embodiment of the present application. Fig.14 The main control device 1400 includes a processor 1410 , a memory 1420 , a communication interface 1430 , and at least one communication bus for connecting the processor 1410 , the memory 1420 , and the communication interface 1430 .

[0265] The processor 1410 may be one or more central processing units (CPUs). In the case where the processor 1410 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The memory 1420 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a portable read-only memory (CD-ROM), and the memory 1420 is used to store relevant instructions and data. The communication interface 1430 is used to receive and send data.

[0266] The processor 1410 in the main control device 1400 is used to read one or more programs 1421 stored in the memory 1420 for performing the following steps: receiving first information from a first user of a controlled device, where the controlled device is an electronic device in a registered device group determined in response to the device wake-up information of the first user, and the registered device group is composed of electronic devices that register with the main control device; determining electronic devices capable of executing the target service function identified by the first information from the registered device group to obtain a candidate device group; obtaining a user decision parameter of the first user and first status information of a first candidate device, where the first candidate device is a candidate device in the candidate device group, and the first status information is a status information in a status information set; determining a target device from the candidate device group according to the user decision parameter and the status information set; and sending second information to the target device, where the second information is used to control the target device to execute the target service function.

[0267] It should be noted that the specific implementation of each operation performed by the main control device 1400 can refer to the above Figure 6 or Fig.11 The corresponding description of the method embodiment shown will not be repeated here.

[0268] The following is a schematic diagram of a structure of a controlled device provided in an embodiment of the present application. Fig.15 The controlled device 1500 includes a processor 1510 , a memory 1520 , a communication interface 1530 , and at least one communication bus for connecting the processor 1510 , the memory 1520 , and the communication interface 1530 .

[0269] The processor 1510 may be one or more central processing units (CPUs). In the case where the processor 1510 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The memory 1520 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a portable read-only memory (CD-ROM), and the memory 1520 is used to store related instructions and data. The communication interface 1530 is used to receive and send data.

[0270] The processor 1510 in the controlled device 1500 is used to read one or more programs 1521 stored in the memory 1520 to perform the following steps: obtaining first information of the first user, the controlled device is an electronic device in the registered device group determined in response to the device wake-up information of the first user, and the registered device group is composed of electronic devices that register with the main control device; sending first information to the main control device, the first information is used by the main control device to determine the target device that executes the target service function identified by the first information from the registered device group.

[0271] It should be noted that the specific implementation of each operation performed by the controlled device 1500 can refer to the above Figure 6 or Fig.11 The corresponding description of the method embodiment shown will not be repeated here.

[0272] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0273] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute some or all steps of any method described in the above method embodiment. The computer program product may be a software installation package.

[0274] It should be noted that, for the above-mentioned various method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.

[0275] In the above embodiments, the present application has different focuses on the description of each embodiment. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0276] In several embodiments provided in the present application, those skilled in the art should be aware that the described device can be implemented in other ways. It is understandable that the device embodiments described above are merely schematic. For example, the division of the above units is only a logical function division, and there may be other division methods in practice. In other words, multiple units or components can be combined or integrated into another software, and some features can be ignored or not executed. In addition, the coupling, direct coupling or communication connection between each other shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, or can be electrical or other forms.

[0277] If the above-mentioned units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. It is understandable that the technical solution of the present application (the part of the technical solution that contributes to the prior art or all or part of the technical solution) can be embodied in the form of a computer software product. The computer software product is stored in a memory, including several instructions for a computer device (a personal computer, a server or a network device, etc.) to perform all or part of the steps of the embodiment of the present application. In addition, the above-mentioned computer-readable storage medium can be stored in various memories such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk or an optical disk.

[0278] The embodiments of the present application are specifically introduced above. Those skilled in the art should be aware that the embodiments of the present application are only used to help understand the core idea of ​​the technical solution of the present application, so the embodiments of the present application may be changed in the specific implementation and scope of application. So far, the contents recorded in this specification should not be understood as limiting the scope of protection of the present application. In addition, on the basis of the technical solution of the embodiments of the present application, any modification, equivalent replacement, improvement, etc. made shall be included in the scope of protection of the embodiments of the present application.

Claims

1. An information processing method, characterized in that: Applied to a main control device; the method comprises: Receiving first information from a first user of a controlled device, the controlled device being an electronic device in a registered device group determined in response to device wake-up information of the first user, the registered device group consisting of electronic devices registered with the master device, the first information including voice control information; Determine a candidate device group from the registered device group, each candidate device in the candidate device group is an electronic device capable of executing the target service function identified by the first information; Acquire a user decision parameter of the first user and first status information of a first candidate device, where the first candidate device is one of the candidate devices in the candidate device group, and the first status information includes at least one of a geographical distance, historical usage time, and historical usage frequency between the first user and the first candidate device and / or at least one of a device status, service status suitability, and device location of the first candidate device; determining a target device from the candidate device group according to the user decision parameter and the first state information; Sending second information to the target device, where the second information is used to control the target device to execute the target service function.

2. The method according to claim 1, characterized in that The determining a target device from the candidate device group according to the user decision parameter and the first state information includes: Determine a first parameter value for the first candidate device according to the user decision parameter and the first state information, the first parameter value being used to indicate a probability of the first candidate device being used to execute the target service function, the first parameter value being one of a parameter value set; The target device is determined from the candidate device group according to the parameter value set and a preset threshold.

3. The method according to claim 2, characterized in that The determining the target device from the candidate device group according to the parameter value set and a preset threshold comprises: If the target parameter value in the parameter value set is greater than or equal to the preset threshold, the candidate device corresponding to the target parameter value is used as the target device, and the target parameter value is the maximum value in the parameter value set; or If the target parameter value is less than the preset threshold, generating a request message for requesting the first user to select a candidate device to be controlled from the candidate device group, and sending the request message to the controlled device; obtaining response information from the controlled device for the request information; and using the candidate device to be controlled indicated by the response information as the target device; or At least one candidate device corresponding to a parameter value in the parameter value set that is greater than or equal to the preset threshold is used as the target device.

4. The method according to claim 3, characterized in that: After obtaining response information from the controlled device to the request information, the method further includes: The user decision parameter is updated according to the response information.

5. The method according to claim 1, characterized in that The step of determining electronic devices capable of executing the target service function identified by the first information from the registered device group to obtain a candidate device group includes: identifying the first information to obtain a target service function; Obtaining device registration information of each electronic device in the registered device group; The candidate device group is obtained by determining electronic devices capable of executing the target service function from the registered device group according to the device registration information.

6. The method according to claim 5, characterized in that The identifying the first information to obtain a target service function includes: If the first information includes the voice control information, the target service function is determined according to the word vector matrix identified by the voice control information and a pre-trained service function analysis model; or, If the first information includes gesture control information, the target service capability is determined according to the target gesture type identified in the gesture control information and a preset mapping relationship, where the preset mapping relationship is used to indicate a mapping relationship between the target gesture type and the target service capability.

7. The method according to any one of claims 1 to 6, characterized in that: After sending the second information to the target device, the method further includes: Device adjustment information is received from the target device, where the device adjustment information is used to request the main control device to adjust the electronic device that executes the target service function.

8. An information processing method, characterized in that: Applied to a controlled device; the method comprises: Acquire first information of a first user, wherein the controlled device is an electronic device in a registered device group determined in response to device wake-up information of the first user, the registered device group is composed of electronic devices registered with a master control device, and the first information includes voice control information; The first information is sent to the master control device, where the first information is used by the master control device to determine a target device that executes the target service function identified by the first information from the registered device group, where the target device is determined by the master control device in the following manner: The master control device determines a candidate device group from the registered device group, each candidate device in the candidate device group is an electronic device capable of executing the target service function identified by the first information; the master control device obtains a user decision parameter of the first user and first status information of a first candidate device, the first candidate device is a candidate device in the candidate device group, and the first status information includes at least one of the geographical distance, historical usage time, and historical usage frequency between the first user and the first candidate device and / or at least one of the device status, service status suitability, and device location of the first candidate device; the master control device determines a target device from the candidate device group based on the user decision parameter and the first status information.

9. An information processing device, characterized in that: Applied to a main control device; the device comprises a processing unit and a communication unit, and the processing unit is used to: receiving, by the communication unit, first information from a first user of a controlled device, the controlled device being an electronic device in a registered device group determined in response to device wake-up information of the first user, the registered device group consisting of electronic devices registered with the master control device; Determining electronic devices capable of executing the target service function identified by the first information from the registered device group to obtain a candidate device group; Acquire a user decision parameter of the first user and first state information of a first candidate device, where the first candidate device is a candidate device in the candidate device group, and the first state information is a state information in a state information set; Determining a target device from the candidate device group according to the user decision parameter and the state information set; The second information is sent to the target device through the communication unit, where the second information is used to control the target device to execute the target service function.

10. An information processing device, characterized in that: Applied to a controlled device; the device comprises a processing unit and a communication unit, and the processing unit is used to: Acquire first information of a first user, wherein the controlled device is an electronic device in a registered device group determined in response to device wake-up information of the first user, and the registered device group is composed of electronic devices registered with a master control device; The first information is sent to the main control device through the communication unit, and the first information is used by the main control device to determine a target device that executes the target service function identified by the first information from the registered device group.

11. A main control device, characterized in that: The main control device includes an electronic device or a server; the main control device includes a processor, a memory and a communication interface, the memory stores one or more programs, and the one or more programs are executed by the processor, and the one or more programs include instructions for executing the steps in the method described in any one of claims 1-7.

12. A controlled device, characterized in that: The controlled device includes an electronic device; the controlled device includes a processor, a memory and a communication interface, the memory stores one or more programs, and the one or more programs are executed by the processor, and the one or more programs include instructions for executing the steps in the method as claimed in claim 8.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program is operable to cause a computer to execute the method according to any one of claims 1 to 8.

Images