Display apparatus and control method
Patent Information
- Application Number
- CN202380027495.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-30
- Filing Date
- 2023-02-24
- Publication Date
- 2024-11-15
AI Technical Summary
When the existing display device receives the user's voice control information, it needs to send the information to the server for semantic analysis, which results in excessive server resource consumption and increases server pressure.
After receiving the user's target control request, the display device performs semantic matching through the preset text library and directly executes the control instructions, avoiding the need to upload to the server for semantic analysis every time there is an interaction.
It effectively saves server resources, improves the control efficiency of user interaction, reduces the number of visits to the server, and reduces the pressure on the server.
Smart Images

Figure CN118974823A_ABST
Abstract
Description
Display device and control method
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to application number 202210713044.3 filed on June 22, 2022, and application number 202210768515.0 filed on June 30, 2022, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The embodiments of the present application relate to the field of display technology, and in particular, to a display device and a control method. Background Art
[0004] Current display devices can support user voice interaction, and users can control the display device through voice, such as voice controlling the volume of the display device, voice searching for relevant information in the display device, etc.
[0005] When a display device receives voice control information from a user, it needs to send the voice control information to the server for semantic analysis, and can only respond to the user's voice information based on the semantic analysis results returned by the server. However, every interaction between the user and the display device requires the server to perform semantic analysis, which will increase the pressure on the server and cause a large loss of server resources.
[0006] Summary of the Invention
[0007] An embodiment of the present application provides a display device, which includes: a controller, configured to: receive a user's target control request for the display device; in response to the target control request, obtain a target control text, perform semantic matching on the target control text based on a preset text library to obtain a semantic matching result, wherein the preset text library includes: preset terminology information and preset resource information; and based on the semantic matching result, execute a control instruction corresponding to the target control request.
[0008] An embodiment of the present application provides a control method, including: receiving a target control request from a user for the display device; in response to the target control request, obtaining a target control text, performing semantic matching on the target control text based on a preset text library to obtain a semantic matching result, wherein the preset text library includes: preset terminology information and preset resource information; based on the semantic matching result, executing a control instruction corresponding to the target control request; and displaying a response interface corresponding to the execution of the control instruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] FIG1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to one or more embodiments;
[0010] FIG2 is a block diagram of a hardware configuration of a control device 100 according to one or more embodiments;
[0011] FIG3 is a block diagram of a hardware configuration of a display device 200 according to one or more embodiments;
[0012] FIG4 is a schematic diagram of software configuration in the display device 200 according to one or more embodiments;
[0013] FIG5 is a schematic diagram showing an icon control interface of an application in a display device 200 according to one or more embodiments;
[0014] FIG6A is a system framework diagram for performing control according to one or more embodiments;
[0015] FIG6B is a diagram illustrating an architecture for performing control according to one or more embodiments;
[0016] FIG7A is a schematic flow chart of a control method;
[0017] FIG7B is a schematic diagram of an interface of a display device;
[0018] FIG7C is a schematic diagram of an interface of another display device;
[0019] FIG7D is a schematic diagram of an interface of yet another display device;
[0020] FIG7E is a schematic diagram of an interface of another display device;
[0021] FIG7F is a schematic diagram of an interface of another display device;
[0022] FIG7G is a schematic diagram of an interface of another display device;
[0023] FIG8A is a schematic diagram showing a structure of matching a target control text with candidate terms;
[0024] FIG8B is a schematic diagram showing another structure of matching target control text with candidate terms;
[0025] FIG8C is a schematic diagram showing another structure of matching target control text with candidate terms;
[0026] FIG8D is a schematic diagram showing a structure of another matching target control text and candidate terms;
[0027] FIG8E is a schematic diagram showing a structure of another matching method between a target control text and a candidate term;
[0028] FIG8F is a schematic diagram of a structure of communication between a display device and a server;
[0029] FIG9 is a schematic diagram of an operation scenario between a display device and a control apparatus according to one or more embodiments;
[0030] FIG10A is a flow chart of a semantic understanding method;
[0031] FIG10B is a schematic diagram of preprocessing of a semantic understanding method;
[0032] FIG11A is a flow chart of another semantic understanding method;
[0033] FIG11B is a schematic diagram showing the principle of another semantic understanding method;
[0034] FIG12A is a flow chart of another semantic understanding method;
[0035] FIG12B is a schematic diagram showing the principle of yet another semantic understanding method;
[0036] FIG12C is one of the interface display diagrams of a semantic understanding method;
[0037] FIG12D is a second interface display diagram of a semantic understanding method;
[0038] FIG12E is a third interface display diagram of a semantic understanding method. DETAILED DESCRIPTION
[0039] In order to more clearly understand the above-mentioned purposes, features and advantages of the embodiments of the present application, the embodiments of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the embodiments of the present application and the features therein can be combined with each other.
[0040] In the following description, many specific details are set forth to facilitate a full understanding of the embodiments of the present application, but the embodiments of the present application can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, not all of the embodiments.
[0041] The display device can respond to the user's control instructions in the display interface by interacting with the user. The user can send voice control information to the display device to perform corresponding voice control on the display device. After receiving the voice control information sent by the user, the display device will send the voice control information to the server for semantic analysis, so that the corresponding control can be performed based on the semantic analysis results returned by the server.
[0042] However, the same user may have many fixed language habits. For example, when controlling the volume of a display device, some users like to say "increase the volume" while others like to say "louder volume." For fixed users, the way each user expresses their control needs will not change much. Every time the display device receives a user's voice control information, it needs to perform voice analysis on the server side, which wastes server resources.
[0043] In order to solve the above problems, in the embodiment of the present application, after receiving the target control request for the display device sent by the user, the display device performs text conversion on the target control request to obtain the target control text, performs semantic matching on the target control text based on the preset text library, determines the semantic matching result, and can realize semantic understanding when interacting with the user on the device side, and based on the semantic matching result, executes the control instruction corresponding to the target control request, and displays the response interface corresponding to the execution control instruction. Through the above method, semantic matching can be performed on the device side based on the preset text library, avoiding the problem of uploading to the server and consuming server resources every time the user interacts, which can effectively save the pressure on the server and improve the efficiency of interactive control.
[0044] Figure 1 is a schematic diagram of an operational scenario between a display device and a control device according to one or more embodiments. As shown in Figure 1, a user can speak voice control information to the display device 200, or the user can send voice control information to the control device 100 of the display device 200 (or the smart device 300 associated with the display device 200) to implement semantic control of the display device. The display device 200 performs semantic matching locally on the device side, effectively recognizes the voice control information sent by the user, executes the relevant control instructions, and displays the response interface corresponding to the executed control instructions, thereby facilitating the user experience.
[0045] In some embodiments, the user can turn on the display device with a remote control or a mobile phone and send voice control information to the display device. The display device can perform semantic matching on the voice control information and identify the user's needs.
[0046] Exemplarily, a user can send voice data to the display device at a viewing position of the display device, and the microphone array of the display device collects the voice data spoken by the user and performs semantic matching. Specifically, a plurality of preset positions can be set on the display device to load the microphone array, so as to effectively obtain external voice data. For example, a plurality of preset positions can be set at the bottom of the display of the display device, such as a first position, a second position and a third position, a microphone array can be loaded in the first position, a microphone array can be loaded in the second position, and a microphone array can be loaded in the third position. Alternatively, a plurality of preset positions can be set on the upper part of the display of the display device, such as a fourth position, a fifth position and a sixth position, a microphone array can be loaded in the fourth position, a microphone array can be loaded in the fifth position, and a microphone array can be loaded in the sixth position. Alternatively, a plurality of preset positions can be set on the display of the display device, such as a fourth position, a fifth position and a sixth position, a microphone array can be loaded in the fourth position, a microphone array can be loaded in the fifth position, and a microphone array can be loaded in the sixth position. Alternatively, a plurality of preset positions can be set on the display of the display device Multiple preset positions are set on the left side, such as the seventh position, the eighth position and the ninth position, and a microphone array can be loaded in the seventh position, a microphone array can be loaded in the eighth position, and a microphone array can be loaded in the ninth position. Alternatively, multiple preset positions can be set on the right side of the display of the display device, such as the tenth position, the eleventh position and the twelfth position, and a microphone array can be loaded in the tenth position, a microphone array can be loaded in the eleventh position, and a microphone array can be loaded in the twelfth position. When the display device receives the user's voice data, it can effectively and quickly perform semantic matching to respond to the user, thereby solving the problems in the prior art. It can perform semantic matching based on the preset text library on the device side, avoiding the problem of uploading to the server and consuming server resources every time the user interacts, which can effectively save server pressure and improve interactive control efficiency.
[0047] It should be noted that the loading position of the microphone array in the display device can be installed in different areas of the display at the same time. The first area, the second area,..., the twelfth area can exist at the same time, so that voice data can be obtained more effectively and accurately. The embodiments of the present application do not make specific limitations on this.
[0048] In some embodiments, the control device 100 may be a remote controller, and communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, and control panel input. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, or other smart device may also be used to control the display device 200.
[0049] In some embodiments, the display device 200 may not use the above-mentioned smart device or control device to receive instructions, but may receive user control through touch, gestures, or voice input.
[0050] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.
[0051] In some embodiments, the smart device 300 can connect and communicate with the software application installed in the display device 200 through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the smart device 300 can also be transmitted to the display device 200 to achieve a synchronous display function. The display device 200 also communicates data with the server 400 through a variety of communication methods, allowing the display device 200 to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 can provide various content and interactions to the display device 200. The display device 200 can be a liquid crystal display, an OLED display or a projection display device, etc. In addition to providing a broadcast receiving television function, the display device 200 can also provide an additional intelligent network TV function that provides computer support functions.
[0052] FIG2 is a block diagram of a configuration of a control device 100 according to an exemplary embodiment. As shown in FIG2 , the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation instructions and convert the operation instructions into instructions that can be recognized and responded to by the display device 200, thereby acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, NFC, or an alternative module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, a button, or an alternative module.
[0053] FIG3 is a block diagram of the hardware configuration of a display device 200 according to an exemplary embodiment. As shown in FIG3 , the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface (i.e., a user input interface) 280. The controller 250 includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output. The display 260 may be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and may also be a projection device and a projection screen. The tuner 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example: the communicator may include a Wifi module, a Bluetooth module, a wired Ethernet module or other network communication protocol chips or near-field communication protocol chips, and at least one of an infrared receiver. The display device 200 can establish the sending and receiving of control signals and data signals with the external control device 100 or the server 400 through the communicator 220. The detector 230 is used to collect signals from the external environment or interactions with the outside. The controller 250 and the tuner and demodulator 210 can be located in different split devices, that is, the tuner and demodulator 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box. The user interface 280 can be used to receive control signals from the control device 100 (such as an infrared remote control, etc.).
[0054] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. The user can enter user commands through the graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the graphical user interface (GUI). Alternatively, the user can enter user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.
[0055] Figure 4 is a schematic diagram of the software configuration in the display device 200 according to one or more embodiments of the present application. As shown in Figure 4, the system is divided into four layers, from top to bottom, namely the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.
[0056] In some embodiments, at least one application program runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the applications in the application layer include but are not limited to the above examples.
[0057] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.
[0058] In some embodiments, the kernel layer is a layer between hardware and software, and includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.
[0059] Figure 5 is a schematic diagram of the icon control interface display of an application in the display device 200 according to one or more embodiments of the present application. As shown in Figure 5, the application layer includes at least one application that can display a corresponding icon control in the display, such as: a live TV application icon control, a video on demand application icon control, a media center application icon control, an application center icon control, a game application icon control, etc. The live TV application can provide live TV through different signal sources. The video on demand application can provide videos from different storage sources. Unlike the live TV application, the video on demand provides video display from certain storage sources. The media center application can provide applications for playing various multimedia content. The application center can provide storage for various applications.
[0060] In some embodiments, the display device is a terminal device with a display function, such as a television or a flat-panel TV. In the display device:
[0061] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. The user can enter user commands through the graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the graphical user interface (GUI). Alternatively, the user can enter user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.
[0062] an output interface (display 260 and / or audio output interface 270 ), configured to output user interaction information;
[0063] The communicator 220 is used to communicate with the server 400;
[0064] The controller 250 is configured to: receive a target control request from a user for the display device;
[0065] In response to the target control request, a target control text is obtained, and semantic matching is performed on the target control text based on a preset text library to obtain a semantic matching result, wherein the preset text library includes: preset terminology information and preset resource information;
[0066] Based on the semantic matching result, executing the control instruction corresponding to the target control request;
[0067] The display 260 is configured to display a response interface corresponding to the execution of the control instruction.
[0068] In some embodiments, the controller 250 is specifically configured to:
[0069] Performing semantic matching on the target control text based on preset terminology information;
[0070] It is detected that the target control text does not match any of the terms included in the preset term information, and semantic matching is performed on the target control text based on preset resource information to obtain a semantic matching result.
[0071] In some embodiments, the controller 250 is specifically configured to:
[0072] detecting that the target control text matches a candidate term included in the preset term information, and determining that the control instruction corresponding to the target control request is a first control instruction corresponding to the candidate term;
[0073] Execute a first control instruction corresponding to the candidate term.
[0074] In some embodiments, the controller 250 is specifically configured to:
[0075] detecting that the target control text matches a candidate resource included in the preset resource information, and determining that the control instruction corresponding to the target control request is a second control instruction corresponding to the candidate resource;
[0076] Execute the second control instruction corresponding to the candidate resource.
[0077] In some embodiments, the controller 250 is further configured to:
[0078] In response to the first information sending request, receiving preset terminology information sent by the server, wherein the preset terminology information is determined by the server based on the user's historical control requests;
[0079] The preset term information is added to the preset text library.
[0080] In some embodiments, the controller 250 is further configured to:
[0081] In response to the second information sending request, receiving preset resource information sent by the server, wherein the preset resource information is determined by the server based on the user's historical control requests and the user's historical page views, and the preset resource information includes popular resources in at least two fields;
[0082] The preset resource information is added to the preset text library.
[0083] In some embodiments, the controller 250 is specifically configured to:
[0084] detecting that the target control text does not match any of the resources included in the preset resource information, sending the target control text to a server so that the server performs semantic analysis on the target control text and determines a third control instruction corresponding to the target control text;
[0085] In response to a third control instruction corresponding to the target control text sent by the server, the third control instruction is executed.
[0086] To sum up, in the embodiment of the present application, after receiving the target control request for the display device sent by the user, the display device performs text conversion on the target control request to obtain the target control text, performs semantic matching on the target control text based on the preset text library, determines the semantic matching result, and can realize semantic understanding when interacting with the user on the device side, and based on the semantic matching result, executes the control instruction corresponding to the target control request, and displays the response interface corresponding to the execution control instruction. Through the above method, semantic matching can be performed on the device side based on the preset text library, avoiding the problem of uploading to the server and consuming server resources every time the user interacts, which can effectively save the pressure on the server and improve the efficiency of interactive control.
[0087] FIG6A is a system framework diagram for control according to one or more embodiments of the present application. As shown in FIG6A , the system may include a control request receiving module 601, a semantic matching module 602, a control instruction execution module 603, and a response interface display module 604. The system receives a user's target control request for the display device through the control request receiving module 601. The semantic matching module 602 responds to the target control request, obtains a target control text, and performs semantic matching on the target control text based on a preset text library to obtain a semantic matching result. The preset text library includes: preset terminology information and preset resource information. The control instruction execution module 603 executes the control instruction corresponding to the target control request based on the semantic matching result, and displays the response interface corresponding to the execution of the control instruction through the response interface display module 604. Thus, semantic matching can be performed on the device side based on the preset text library, avoiding the problem of uploading to the server and consuming server resources every time the user interacts, effectively saving server pressure and improving interactive control efficiency.
[0088] Figure 6B is an architecture diagram for control according to one or more embodiments of the embodiments of the present application. Based on the above-mentioned system framework, the implementation of the embodiments of the present application in the Android system is shown in Figure 6B. The Android system mainly includes an application layer, a framework layer, a system runtime layer and a kernel layer. The implementation logic is mainly reflected in the application layer, which includes a control request receiving module, a semantic matching module, a control instruction execution module and a response interface display module.
[0089] The control method provided in the embodiment of the present application detects the user's voice behavior in real time, receives the voice data sent by the user, collects the user's voice data within a speaking period as the user's target control request for the display device, and performs text analysis on the target control request to obtain the target control text, wherein the display device can pre-store a preset text library, and the preset text library can be used to describe the user's habitual terms and preset resource information for describing current popular resources. The target control text is similarly matched by the preset term information and the preset resource information to determine the semantic matching result, so that according to the semantic matching result, the corresponding control instruction is executed on the display device to control the display device, and the user interface corresponding to the execution of the control instruction is displayed, so that the user can know the control result. Therefore, semantic matching can be performed on the device side, and the server does not need to be involved in each interaction, which reduces the number of visits to the server, avoids the problem of resource loss caused by a large number of invalid visits on the server, reduces the pressure on the server, and improves the interaction efficiency.
[0090] The following will be explained in an exemplary manner in conjunction with FIG. 7A . It is understandable that the steps involved in FIG. 7A may include more steps or fewer steps in actual implementation, and the order of these steps may also be different.
[0091] As shown in FIG7A , FIG7A is a flow chart of a control method provided in an embodiment of the present application. The control method specifically includes the following steps:
[0092] S710: Receive a target control request from a user for a display device.
[0093] Among them, the display device may include some smart devices capable of voice interaction, such as, the display device may include: smart TV, smart air conditioner, smart refrigerator, smart speaker, etc. Of course, the display device may also include other smart devices that support voice interaction, and the embodiments of this application do not make specific limitations on this.
[0094] Among them, the display device can record the interactive voice emitted by the user in real time to feedback the interactive information of the interactive voice response to the user.
[0095] When the display device is a smart TV, the interactive voice between the user and the smart TV may be "turn down the volume". Correspondingly, the user's target control request for the smart TV is "turn down the volume", indicating that the user has a need to lower the volume. The smart TV can effectively determine the user's target control request for the smart TV by collecting the user's voice data in real time.
[0096] Among them, the smart TV can collect the voice data of users in its corresponding preset area in real time. If no voice data is detected within the preset period, or the detected voice data is unclear, the preset area can be expanded for continuous voice collection, thereby effectively and accurately collecting the user's voice data.
[0097] It should be noted that the preset area may be an area corresponding to a preset sound field range of the smart TV, or other areas around the preset smart TV, and the embodiment of the present application does not make specific limitations on this.
[0098] When the display device is a smart air conditioner, the interactive voice between the user and the smart air conditioner may be "lower the temperature". Correspondingly, the user's target control request for the smart air conditioner is "lower the temperature", indicating that the user has a need to lower the temperature. The smart air conditioner can effectively determine the user's target control request for the smart air conditioner by collecting the user's voice data in real time.
[0099] Among them, the smart air conditioner can collect the voice data of users in its corresponding preset area in real time. If no voice data is detected within the preset period, or the detected voice data is unclear, the preset area can be expanded for continuous voice collection, thereby effectively and accurately collecting the user's voice data.
[0100] It should be noted that the preset area may be other areas around the pre-set smart air conditioner, such as a designated area in front of the smart air conditioner.
[0101] When the display device is a smart refrigerator, the interactive voice between the user and the smart refrigerator may be "open / on", and correspondingly, the user's target control request for the smart refrigerator is "open / on", indicating that the user has the need to open the refrigerator. The smart refrigerator can effectively determine the user's target control request for the smart air conditioner by collecting the user's voice data in real time.
[0102] Among them, the smart refrigerator can collect the voice data of users in its corresponding preset area in real time. If no voice data is detected within the preset period, or the detected voice data is unclear, the preset area can be expanded for continuous voice collection, thereby effectively and accurately collecting the user's voice data.
[0103] It should be noted that the preset area may be other pre-set areas around the smart refrigerator, such as a designated area in front of the smart refrigerator, and the embodiment of the present application does not make specific limitations on this.
[0104] When the display device is a smart speaker, the interactive voice between the user and the smart speaker may be "increase the playback volume". Correspondingly, the user's target control request for the smart speaker is "increase the playback volume", indicating that the user has a need to increase the playback volume. The smart speaker can effectively determine the user's target control request for the smart speaker by collecting the user's voice data in real time.
[0105] Among them, the smart speaker can collect the voice data of users in its corresponding preset area in real time. If no voice data is detected within the preset period, or the detected voice data is unclear, the preset area can be expanded for continuous voice collection, thereby effectively and accurately collecting the user's voice data.
[0106] It should be noted that the preset area may be an area corresponding to the sound field range of the preset smart speaker, or other areas around the preset smart speaker.
[0107] Among them, for smart speakers, the preset area may be related to the current placement position of the smart speaker. For example, when the smart speaker moves from a first position to a second position, the preset area may be changed from a first preset area corresponding to the first position to a second position area corresponding to the second position.
[0108] S720 : In response to the target control request, obtain a target control text, perform semantic matching on the target control text based on a preset text library, and obtain a semantic matching result.
[0109] The display device may perform a text conversion operation on the received target control request, converting the voice data into text data, so as to facilitate the display device to perform text matching.
[0110] It should be noted that the text conversion operation is an operation of converting voice data into text data. Specifically, the text conversion operation can be implemented through a voice recognition algorithm / voice recognition software.
[0111] Among them, the display device can perform semantic matching on the target control text based on the preset text library on the device side, thereby avoiding multiple visits to the server side and occupying server resources.
[0112] The preset text library may include: preset term information and preset resource information. The preset user information is the user's usual idioms, and the preset resource information is the popular words on the network / online in the current period.
[0113] The preset terminology information may be the words that the user habitually uses when interacting with the display device by voice. When the user interacts with the smart TV by voice, the preset user information may include but is not limited to: turn on the TV, turn on the TV, lower the volume, increase the volume, turn off the TV, etc. The preset resource information may be: song name A, TV series B, video C, etc.
[0114] When a user interacts with a smart air conditioner through voice, the preset user information may include but is not limited to: turn on the air conditioner, turn on the air conditioner, lower the temperature, increase the temperature, turn off the air conditioner, etc., and the preset resource information may be: 26°, 16°, etc.
[0115] When a user interacts with a smart refrigerator through voice, the preset user information may include but is not limited to: opening the refrigerator, turning on the refrigerator, lowering the temperature, increasing the temperature, closing the refrigerator, etc., and the preset resource information may be: silent mode, deodorization mode, etc.
[0116] When a user interacts with a smart speaker through voice, the preset user information may include but is not limited to: turn on the speaker, turn on the speaker, increase the volume, decrease the volume, turn off the speaker, etc. The preset resource information may be: song name D, pop music E, etc.
[0117] In addition, the preset text library can correspond to the user, that is, the preset terminology information and preset resource information corresponding to different users are different. Different users can be identified by the user's voice characteristics, and a preset text library uniquely corresponding to the user can be set, so that when the user's voice data is collected, corresponding matching can be performed based on the corresponding preset text library.
[0118] When it is determined that the person interacting with the display device is the first user, semantic matching can be performed on the target control text based on a first preset text library corresponding to the first user to obtain a semantic matching result, or when it is determined that the person interacting with the display device is the second user, semantic matching can be performed on the target control text based on a second preset text library corresponding to the second user to obtain a semantic matching result.
[0119] Among them, the display device can store preset text libraries corresponding to multiple users. Specifically, the number of preset text libraries that can be stored in the display device is related to the storage capacity of the display device. For example, a display device with a larger storage capacity can store more preset text libraries, and a display device with a smaller storage capacity can store fewer preset text libraries.
[0120] It should be noted that the display device may update the preset text library stored therein according to the storage time or the number of user visits.
[0121] The display device may pre-set an update time (such as one week), and may select to regularly clean up the stored preset text libraries on one Monday of each week, such as deleting some preset text libraries that have not been matched for a long time to facilitate the addition of other new preset text libraries.
[0122] Among them, the display device can be updated regularly and adaptively based on its storage capacity. For example, if the display device stores preset text libraries corresponding to ten users, when it detects that an eleventh preset text library has been added, one preset text library can be deleted according to the number of times the preset text library has been used / the usage time, so as to facilitate the addition of the new preset text library, thereby improving the practicality of the preset text library stored in the display device.
[0123] The display device can record the number of visits of each user. For example, if the number of visits of the first user is 2, the number of visits of the second user is 10, and the number of visits of the third user is 15, it can be determined that the first user is not a frequent user. When the update time arrives, the preset text library corresponding to the first user can be deleted, thereby facilitating real-time updating of the preset text library and ensuring its practicality.
[0124] S730: Execute the control instruction corresponding to the target control request based on the semantic matching result.
[0125] Among them, the semantic matching result can effectively reflect the execution content corresponding to the target control instruction issued by the user, thereby effectively determining the control instruction corresponding to the execution content and executing the control instruction.
[0126] In combination with the above example, when the display device is a smart TV, when it is determined based on the semantic matching result that the execution content corresponding to the target control instruction issued by the user is to increase the volume, the control instruction corresponding to the execution target control request is to increase the current playback volume of the smart TV.
[0127] When the display device is a smart air conditioner, when it is determined based on the semantic matching result that the execution content corresponding to the target control instruction issued by the user is to lower the temperature, the control instruction corresponding to the execution target control request is to lower the temperature of the smart air conditioner.
[0128] When the display device is a smart refrigerator, when it is determined based on the semantic matching result that the execution content corresponding to the target control instruction issued by the user is to open the refrigerator door, the control instruction corresponding to the execution target control request is to open the door of the smart refrigerator.
[0129] When the display device is a smart speaker, when it is determined based on the semantic matching result that the execution content corresponding to the target control instruction issued by the user is to increase the volume, the control instruction corresponding to the execution target control request is to increase the playback volume of the smart speaker.
[0130] In some embodiments, it also includes: displaying a response interface corresponding to the execution control instruction.
[0131] Among them, the response interface can be a newly added display content on the original interface, or it can be a display interface after the original interface is modified. The response interface corresponding to the display execution control instruction can be divided into the following multiple implementation methods.
[0132] In some embodiments, when executing the control instruction corresponding to the target control request does not change the current display interface of the display device, the response interface corresponding to the execution control instruction may be a display interface corresponding to adding new display content to the current display interface of the display device.
[0133] When the display device is a smart TV, the current display interface of the smart TV plays display content 1, as shown in Figure 7B. The control instruction corresponding to the target control request is to turn up the volume. When the control instruction corresponding to the target control request is executed, the current display interface of the smart device will not change. The response interface corresponding to the corresponding display execution control instruction can be a display indication for turning up the volume in a preset area of the current display interface. The response interface is shown in Figure 7C.
[0134] When the display device is a smart refrigerator, the current display interface of the smart refrigerator shows that the temperature of the refrigerator compartment is 5°C, as shown in Figure 7D. The control instruction corresponding to the target control request is to adjust the temperature of the refrigerator compartment to 1°C. When the control instruction corresponding to the target control request is executed, the current display interface of the smart device will not change. The response interface corresponding to the corresponding display execution control instruction can be to adjust the displayed temperature to 1°C in the temperature display area of the current display interface, and can display an adjusted message. The response interface is shown in Figure 7E.
[0135] In other embodiments, when executing a control instruction corresponding to a target control request changes a current display interface of a display device, a response interface corresponding to the displayed execution control instruction may be another display interface after the current display interface of the display device has been changed.
[0136] When the display device is a smart TV, the current display interface of the smart TV displays the first episode of TV series 1, and the playback content is display content 2, as shown in Figure 7F. The control instruction corresponding to the target control request is to play the next episode. When the control instruction corresponding to the target control request is executed, the current display interface of the smart TV will change to another display interface. The response interface corresponding to the corresponding display execution control instruction can be the display interface corresponding to the next episode after the current display interface changes, displaying the second episode of TV series 1, and the playback content is display content 3. The response interface is shown in Figure 7G.
[0137] Based on the description of the above embodiment, when determining the semantic matching result, the target control text can be semantically matched according to the preset term information and the preset resource information in the preset text library in turn. For example, all the candidate terms included in the preset term information are first semantically matched with the target control text. If none of the candidate terms included in the preset term information can be successfully matched with the target control text, all the candidate resources included in the preset resource information can be selected to match the target control text one by one, thereby determining the semantic matching result.
[0138] Among them, a matching threshold can be set to measure the matching degree between the target control text and each candidate term included in the preset term information. For example, the matching threshold can be set to 85%. If the matching degree between the target control text and the candidate terms included in the preset term information is greater than 85%, the match is determined to be successful. If the matching degree between the target control text and the candidate terms included in the preset term information is less than or equal to 85%, the match is determined to have failed.
[0139] As exemplified in FIG8A , the target control text is matched with the five candidate terms included in the preset term information, wherein the target control text matches candidate term 1 with a degree of 25%, the target control text matches candidate term 2 with a degree of 28%, the target control text matches candidate term 3 with a degree of 40%, the target control text matches candidate term 4 with a degree of 50%, and the target control text matches candidate term 5 with a degree of 10%.
[0140] Among them, when it is determined that the target control text matches a candidate term in the preset term information (the matching degree is higher than 85%), it can be determined that the control instruction corresponding to the target control request is the same as the first control instruction corresponding to this candidate term, and the first control instruction can be executed to respond to the target control request.
[0141] As exemplified in FIG8B , the target control text is matched with the five candidate terms included in the preset term information, wherein the matching degree between the target control text and candidate term 1 is 25%, the matching degree between the target control text and candidate term 2 is 90%, the matching degree between the target control text and candidate term 3 is 40%, the matching degree between the target control text and candidate term 4 is 50%, and the matching degree between the target control text and candidate term 5 is 10%. It can be determined that the candidate term matching the target control text is candidate term 2.
[0142] It should be noted that if it is determined that at least two candidate terms have a matching degree with the target control text higher than 85%, the candidate term with the highest matching degree may be selected as the candidate term that successfully matches the target control text.
[0143] As exemplified in FIG8C , the target control text is matched with the five candidate terms included in the preset term information, wherein the matching degree between the target control text and candidate term 1 is 25%, the matching degree between the target control text and candidate term 2 is 90%, the matching degree between the target control text and candidate term 3 is 92%, the matching degree between the target control text and candidate term 4 is 50%, and the matching degree between the target control text and candidate term 5 is 10%. It can be determined that the candidate term matching the target control text is candidate term 3.
[0144] Among them, when matching the target control text based on the preset resource information, all candidate resources in the preset resource information can be matched with the target control text. If there is a candidate resource among all the candidate resources included in the preset resource information that can successfully match the target control text, it can be determined that the control instruction corresponding to the target control request is the same as the second control instruction corresponding to this candidate resource, and the second control instruction can be executed to respond to the target control request.
[0145] Among them, a matching threshold can be set to measure the matching degree between the target control text and each candidate resource included in the preset resource information. For example, the matching threshold can be set to 85%. If the matching degree between the target control text and the candidate resources included in the preset resource information is greater than 85%, the match is determined to be successful. If the matching degree between the target control text and the candidate resources included in the preset resource information is less than or equal to 85%, the match is determined to have failed.
[0146] As shown in Figure 8D, the target control text is matched with the three candidate resources included in the preset resource information, among which the matching degree between the target control text and candidate resource 1 is 40%, the matching degree between the target control text and candidate resource 2 is 93%, and the matching degree between the target control text and candidate resource 3 is 20%. It can be determined that the candidate resource matching the target control text is candidate term 2.
[0147] It should be noted that, if it is determined that at least two candidate resources have a matching degree with the target control text higher than 85%, the candidate resource with the highest matching degree may be selected as the candidate resource that successfully matches the target control text.
[0148] As shown in Figure 8E, the target control text is matched with the three candidate resources included in the preset resource information, among which the matching degree between the target control text and candidate resource 1 is 90%, the matching degree between the target control text and candidate resource 2 is 86%, and the matching degree between the target control text and candidate resource 3 is 20%. It can be determined that the candidate resource matching the target control text is candidate term 1.
[0149] The server may predetermine a preset text library based on the user's historical access data, and send the preset text library to the display device for storage.
[0150] The server can send a first information sending request to the display device based on the update frequency of the preset language information, so that the display device receives the preset language information sent by the server to build / update the preset text library.
[0151] Thus, when the display device interacts with the user, it can understand the user's intention based on the preset language information at the device end and quickly give feedback to the user.
[0152] Among them, when determining the preset language information, the server can count the user's habitual expressions according to the user voice access historical data, and the display form of the user access historical data is shown in Table 1 below.
[0153] Table 1 User access historical data
[0154] Among them, the result A is: {"msg":"","slots":[{"name":"singer","value":"Singer 1"},{"name":"song","value":"Song a"}],"code":0,"session_complete":true,"domain":"music","skill_id":"990835315751129088","intent":"play"}.
[0155] Among them, the user's high-frequency statements can be obtained through grouped queries. To improve the accuracy, the grouping conditions can be that the total number of user accesses >= 1000, the number of times of a certain statement >= 30, and the first 20 habitual expressions of each user are obtained.
[0156] Obtain the hash value of this user ID, then take the modulo of 7 (assuming data is pushed between 1-6 am) to obtain the classification value (slot), and save the data in the corresponding Table 2 as shown in Table 2 below.
[0157] Table 2 Classification value of user access historical data
[0158] The server can send a second information sending request to the display device based on the update frequency of the preset resource information, so that the display device receives the preset resource information sent by the server to build / update the preset text library.
[0159] Thus, when the display device interacts with the user, it can understand the user's intention based on the preset resources at the device end and quickly give feedback to the user.
[0160] Among them, when determining the preset resource information, the server can obtain the hot media asset information in the corresponding field according to the user profile. The user profile is the characterization of the user or product feature attributes by collecting data in various dimensions such as the user's social attributes, consumption habits, and preference characteristics, and analyzing and statistically mining the potential value information of these characteristics, so as to abstract the overall picture of a user. The specific process is as follows.
[0161] Create popular resources in each field: Obtain the popular resources in each field according to the search results of all users and the current hot search list, etc., as shown in Table 3 below.
[0162] Table 3 Popular Resources
[0163] Among them, the result B is: {"msg":"","slots":[{"name":"singer","value":"Singer 1"},{"name":"song","value":"Movie"}],"code":0,"session_complete":true,"domain":"music","skill_id":"990836308639354880","intent":"play"}.
[0164] Create a user profile based on the user's voice history access data: Rank the portrait labels of the same user according to the amount of access as shown in Table 4 below.
[0165] Table 4 Portrait Label Ranking
[0166] Obtain popular resources: Obtain the hot media asset results in the corresponding field according to the label type, obtain the hash value of the user identifier, and then take the modulo of 7 (assuming that data is pushed between 1 am and 6 am) to obtain the slot value, and the data is stored in the corresponding Table 5.
[0167] Table 5 Data Storage
[0168] After the server determines the preset term information and preset resource information associated with the user, it can periodically send the preset term information and preset resource information to the display device for the display device to store / update.
[0169] Among them, the user's habitual terms (preset term information) have a smaller range of changes and a low update frequency, and can be set to be updated once a month. The network hot word media information (preset resource information) has a higher update frequency and can be set to be updated once a day. For example, the server can be set to push data between 1:00 and 6:00 a.m. on the 1st of each month according to the slot value in Table 2 at each hour. For example: data with a slot value of 1 is pushed between 1:00 and 2:00, and so on. Similarly, for Table 5, data is pushed between 1:00 and 6:00 a.m. every day according to the slot value.
[0170] In addition, as shown in the example of Figure 8F, the interaction between the display device and the server can adopt the lightweight Message Queuing Telemetry Transport (MQTT) publish / subscribe message transmission protocol. A corresponding topic is established between each display device and the server, and the service quality of QOS1 (a protocol between the data sender and the data receiver) is adopted, and the expiration time is set to 120 seconds. The server side sends network hot word media information and user idiom information to the corresponding topic according to the above procedures. If the corresponding message is consumed successfully, the corresponding data flag position in the database is 1.
[0171] Among them, for special scenarios, such as when the user has not turned on the computer for a long time, the server will only push the user's habitual expressions of the month and the media information of the day's hot words after turning on the computer; if the user fails to read the message, the server will send the message again, and it will reach the display device at least once.
[0172] Among them, MQTT is a machine-to-machine / IoT connection protocol. It is designed as an extremely lightweight publish / subscribe message transmission protocol. It is very useful for remote connections that require a small code footprint and / or where network bandwidth is very valuable. It is designed for constrained devices and low-bandwidth, high-latency or unreliable networks. These principles also make the protocol an ideal choice for connected devices in the emerging "machine-to-machine" or IoT world, as well as mobile applications with very high bandwidth and battery power. It has the advantages of small size, low power consumption, and minimum data packets, and can effectively distribute information to one or more receivers.
[0173] It should be noted that due to the QOS1 transmission protocol adopted by MQTT, the display device will receive duplicate messages. For the duplicate messages received, the display device can adopt the strategy of overwriting the messages for message storage.
[0174] To sum up, the embodiment of the present application executes the above-mentioned control method on the display device. After receiving the target control request for the display device sent by the user, the display device performs text conversion on the target control request to obtain the target control text, performs semantic matching on the target control text based on the preset text library, determines the semantic matching result, and can realize semantic understanding when interacting with the user on the device side, and based on the semantic matching result, executes the control instruction corresponding to the target control request, and displays the response interface corresponding to the execution control instruction. Through the above-mentioned method, semantic matching can be performed on the device side based on the preset text library, avoiding the problem of uploading to the server and consuming server resources every time the user interacts, which can effectively save the pressure on the server and improve the efficiency of interactive control.
[0175] In the above embodiment, the display device is described by taking the voice interaction of a television or the like as an example. In other embodiments of the present application, the display device may also be a display device on a smart refrigerator or other household appliances.
[0176] As shown in Figure 9, Figure 9 is a schematic diagram of an application scenario of the semantic understanding process of a display device provided in an embodiment of the present application. In Figure 9, a user can operate the display device 200 through the control device 100 or the terminal device 300. The semantic understanding process can be used in a voice interaction scenario between a user and a smart home. For example, the display device 200 in this scenario can be a smart device with a smart display function, such as a smart refrigerator or a smart washing machine. When a user wants to control the smart device in this scenario, he needs to issue a voice command first. When the smart device receives the voice command, it performs semantic understanding on the voice command and determines the semantic understanding result corresponding to the voice command, so that the subsequent smart device can display or execute the corresponding control command according to the semantic result, meet the user's usage needs, and improve the user experience.
[0177] In some embodiments, a terminal device 300 (e.g., a mobile terminal, tablet computer, laptop computer, etc.) can also be used to control the display device 200. For example, an application running on the terminal device 300 can be used to control the display device 200. The terminal device 300 can install software applications and connect and communicate with the display device 200 via a network communication protocol, achieving one-to-one control operations and data communication. Semantically understood content displayed on the terminal device 300 can also be transmitted to the display device 200 to achieve a synchronized display function.
[0178] In some embodiments, the display device 200 can also display content based on the user's voice command. For example, if the user's voice command is "Can you help me put two pounds of beef and three pounds of pork with a shelf life of three days into the refrigerator?", the display device 200 will understand the semantics and display the corresponding content.
[0179] In some embodiments, the display device 200 may not receive commands using the terminal device 300 described above, but may instead receive user control through touch or gestures. For example, when the display device 200 is a smart refrigerator and the user adjusts the upper refrigerator door from an open state to a closed state, the display device 200 displays the various ingredients placed in the refrigerator before the door was closed, along with the attribute identifiers corresponding to the ingredients.
[0180] The semantic understanding method provided in the embodiment of the present application can be implemented based on a computer device, or a functional module or functional entity in a computer device.
[0181] The computer device may be a personal computer (PC), a server, a mobile phone, a tablet computer, a laptop computer, a mainframe computer, etc., and the embodiments of the present application do not specifically limit this.
[0182] In order to explain the present embodiment in more detail, the following will be described in conjunction with Figures 10A, 11A, and 12A in an exemplary manner. It is understood that although the steps in the flowcharts of Figures 10A, 11A, and 12A are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in Figures 10A, 11A, and 12A may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps. The semantic understanding method provided in the embodiment of the present application shall prevail. Among them, semantic understanding is to parse text into structured, machine-readable intentions and language slot information through a series of AI (Artificial Intelligence) algorithms, so that Internet developers can better understand and meet user needs. For example, in an embodiment of the present application, the semantic understanding scenario is that the smart refrigerator performs semantic understanding on the multi-ingredient, multi-attribute, or missing-attribute operations spoken by the user, and displays the semantic understanding content on the display device of the smart refrigerator.
[0183] As shown in FIG10A , the method specifically includes the following steps:
[0184] S51. Obtain word segmentation information corresponding to voice ingredient data, where the word segmentation information includes: at least one ingredient segmentation and an attribute identifier corresponding to the ingredient segmentation.
[0185] Specifically, for the voice ingredient data of refrigerator ingredient management, the steps for obtaining the word segmentation information corresponding to the voice ingredient data can be as follows: first, remove prepositions, adverbs and other words that do not have the professional attributes of refrigerator ingredient management; then, obtain all recognizable ingredient names and uniformly mark them according to the type of ingredients; finally, mark other attributes such as quantifiers in different dimensions.
[0186] For example, taking the voice food data "Can you help me put two pounds of beef with a shelf life of 3 days and three pounds of pork in the refrigerator?" as an example, referring to Figure 10B, the corresponding word segmentation information obtained includes: two pounds, shelf life, 3 days, beef, pork, and three pounds.
[0187] S52: Determine a target participle in at least one of the ingredient participles based on the attribute identifier.
[0188] Among them, ingredient segmentation can be mango, apple, beef, pork, mutton, etc., and the attribute identifier corresponding to the ingredient segmentation can be fruit, meat, vegetable, weight, etc. No specific restrictions are imposed on ingredient segmentation and attribute identifiers here.
[0189] For example, in the question "Can you help me put two pounds of beef with a shelf life of three days and three pounds of pork in the refrigerator?", the attribute identifiers are: two pounds, three pounds. The ingredient segmentations determined by this are beef and pork, and the corresponding target segmentations are: beef and pork.
[0190] It should be noted that after preprocessing the user voice data, the corresponding word segmentation information is obtained, and whether there is a subject (ingredient-related words) is determined based on the attribute identifier. If there is a subject, the next step S53 is executed; if there is no subject, the display device can output a prompt such as "I don't understand this sentence" indicating that the keywords of the user voice data cannot be recognized.
[0191] S53: Determine semantic understanding content corresponding to at least one of the ingredient segmentations based on a sequential relationship between the target segmentation and other segmentations in at least one of the ingredient segmentations.
[0192] For example, the semantic understanding content corresponding to the at least one ingredient segmentation is determined based on the order of the target segmentation and other segmentations in the at least one ingredient segmentation. For example, for the sentence "Can you help me put two pounds of beef with a three-day expiration date and three pounds of pork in the refrigerator?", the semantic understanding content corresponding to the at least one ingredient segmentation is determined as: beef, two pounds, expiration date three days; pork, three pounds.
[0193] S54: Display the semantic understanding content corresponding to at least one of the ingredient segmentations.
[0194] For example, the semantic understanding content corresponding to at least one ingredient segmentation is displayed as: beef, two catties, shelf life 3 days; pork, three catties.
[0195] In an embodiment of the present application, the display device first obtains the word segmentation information corresponding to the voice ingredient data, wherein the word segmentation information includes at least one ingredient segmentation and an attribute identifier corresponding to the ingredient segmentation, and then determines the target segmentation in the at least one ingredient segmentation based on the attribute identifier, and then determines the semantic understanding content corresponding to the at least one ingredient segmentation based on the order relationship between the target segmentation and other segmentations in the at least one ingredient segmentation, and finally displays the semantic understanding content corresponding to the at least one ingredient segmentation. By obtaining at least one ingredient segmentation corresponding to the voice ingredient data and an attribute identifier corresponding to the ingredient segmentation, it is possible to understand multiple ingredients and the attributes of each ingredient and their matching relationships, thereby improving the display device's ability to understand inverted and disordered speech in ingredient management, further accurately displaying it, and improving the user experience.
[0196] Figure 11A is a flow chart of another semantic understanding method provided by an embodiment of the present application. This embodiment is a further expansion and optimization based on Figure 10A. Optionally, this embodiment mainly describes the process of step S53 (determining the semantic understanding content corresponding to at least one of the ingredient segmentations based on the sequential relationship between the target segmentation and other segmentations in at least one of the ingredient segmentations).
[0197] S531: Determine phrase information based on an attribute identifier of a first traversal segmentation in at least one of the food segmentations and a sequential relationship between the first traversal segmentation and the target segmentation.
[0198] The target segmentation is the first subject segmentation that is arranged first in the at least one ingredient segmentation. The phrase information includes at least one group of phrases. In this embodiment, the phrase information represents a combination of a phrase corresponding to the first subject segmentation and a phrase corresponding to the second subject segmentation.
[0199] S532: Determine semantic understanding content corresponding to at least one of the ingredient segmentations based on the phrase information.
[0200] For example, referring to FIG11B , taking “Can you help me put two pounds of beef and three pounds of pork with a shelf life of three days into the refrigerator?” as an example, obtain the leftmost subject in the segmentation list, that is, the first subject segmentation is “beef”, and then traverse from the left side of the segmentation list based on the subject. Since the target segmentation is the first subject segmentation with the highest order in at least one ingredient segmentation, in this embodiment, the target segmentation is beef. The phrase corresponding to the first subject segmentation is “two pounds, shelf life of three days”, the second subject segmentation is “pork”, and the phrase corresponding to the second subject segmentation is “three pounds”. Based on the phrase information, the semantic understanding content corresponding to at least one ingredient segmentation is determined to be: beef, two pounds, shelf life of three days; pork, three pounds.
[0201] Figure 12A is a flow chart of another semantic understanding method provided by an embodiment of the present application. This embodiment is a further expansion and optimization based on Figure 6A. Optionally, this embodiment mainly describes the process of step S531 (determining phrase information based on the attribute identifier of the first traversal segmentation in at least one of the food segmentations, and the sequential relationship between the first traversal segmentation and the target segmentation).
[0202] S5311. Determine based on the attribute identifier of the first traversal participle in at least one of the food participles that the first traversal participle is not the second subject participle, and determine that there is no participle with the same attribute identifier as the first traversal participle.
[0203] For example, taking the question "Can you help me put two pounds of beef and three pounds of pork with a shelf life of three days into the refrigerator?", the corresponding segmentation information obtained includes: "two pounds", "shelf life", "3 days", "beef", "pork", and "three pounds". The first segmentation is "two pounds". There can be multiple second subject segmentations. In this embodiment of the application, the second subject segmentation is pork.
[0204] S5312. Determine phrase information based on the sequential relationship between the first traversal word segmentation and the target word segmentation.
[0205] For example, the first traversal word is "two pounds", the target word is "beef", and the phrase information is "shelf life 3 days".
[0206] This embodiment is a further expansion and optimization based on the above embodiment. Optionally, this embodiment mainly describes the process of step S6312 (determining phrase information based on the order relationship between the first traversal segmentation and the target segmentation).
[0207] A. Determine that the order relationship between the first traversal segmentation word and the target segmentation word belongs to a preset relationship, and determine that an object case word exists between the first traversal segmentation word and the target segmentation word.
[0208] For example, taking the question "Can you help me put two pounds of beef and three pounds of pork with a three-day expiration date in the refrigerator?", the corresponding segmentation information obtained includes: "two pounds," "shelf life," "3 days," "beef," "pork," and "three pounds." The first traversal segmentation is "two pounds," and the target segmentation is "beef." The order relationship between the first traversal segmentation and the target segmentation is a preset relationship. There is an object case between the first traversal segmentation and the target segmentation, which is "shelf life."
[0209] B. The object word and the segmentation words corresponding to the object word are combined into a first phrase, and phrase information is determined based on other segmentations between the first traversal segmentation word and the target segmentation word and the first phrase.
[0210] For example, the object word is "shelf life", and the participle corresponding to the object word is "3 days". The first phrase is composed of the object word and the participle corresponding to the object word, that is, "shelf life 3 days".
[0211] Optionally, step B can be implemented as follows:
[0212] B-1. Determine, based on an attribute identifier of a second traversal participle in at least one of the ingredient participles, that the second traversal participle is a second subject participle, and form a second phrase based on other participles between the target participle and the first traversal participle and the first phrase.
[0213] For example, taking the question "Can you help me put two pounds of beef with a three-day shelf life and three pounds of pork in the refrigerator?", the corresponding segmentation information obtained includes: "two pounds," "shelf life," "three days," "beef," "pork," and "three pounds." The second traversal segmentation is "pork," and the attribute identifier of the second traversal segmentation is "three pounds." Therefore, the second traversal segmentation is determined to be the second subject segmentation, and the target segmentation is "beef." The first traversal segmentation is "two pounds." Other segmentations include: the first phrase "shelf life 3 days" and the first subject segmentation "beef."
[0214] The individual participles included in the "other participles" may be the same as the individual participles split from the first phrase (the first phrase may be a compound word). For example, the "other participles" include "shelf life 3 days", and the first phrase is split into "shelf life + 3 days", which forms the second phrase: "beef + 2 pounds + shelf life 3 days".
[0215] Alternatively, the individual participles included in the other participles may be different from the individual participles split out of the first phrase (the first phrase may be a combination word). For example, the other participles may include: "shelf life + 3 days + quality", the first phrase is split into "shelf life + 3 days", and the second phrase formed is "beef + 2 pounds + shelf life 3 days + quality". Generally, beef is divided into 8 levels according to its quality (represented by the marbling of fat) and physiological maturity (age): premium, selected, optional, standard, commercial, available, chopped and canned. In this embodiment, if the other participle is "shelf life + 3 days + premium", the first phrase is split into "shelf life + 3 days", and the second phrase formed is "beef + 2 pounds + shelf life 3 days + premium".
[0216] B-2. Determine that the second traversal participle is the last second subject participle that has not been traversed, and combine the second traversal participle and at least one remaining participle that has not been traversed in the food to form a third phrase.
[0217] For example, the second traversal participle is the last untraversed second subject participle. The second traversal participle is combined with the remaining untraversed participles in at least one of the ingredients to form a third phrase. For example, for the question "Can you help me put two pounds of beef with a three-day shelf life and three pounds of pork in the refrigerator?", the corresponding participle information obtained includes: "two pounds," "shelf life," "three days," "beef," "pork," and "three pounds." The second traversal participle is "pork," and the remaining untraversed participle is "three pounds." The third phrase formed is "pork + three pounds."
[0218] B-3. Determine phrase information based on the sequential relationship between the second phrase and the third phrase.
[0219] Optionally, step B-3 can be implemented as follows:
[0220] ①. Determine candidate information based on the sequential relationship between the second phrase and the third phrase.
[0221] For example, the second phrase is "beef + two pounds + shelf life 3 days", and the third phrase is "pork + three pounds", and the determined candidate information is "beef", "two pounds", "shelf life", "3 days", "pork", and "three pounds".
[0222] ②. Add verb participles to the candidate information to obtain phrase information.
[0223] Exemplarily, as shown in FIG12B , a verb participle is added based on the subject participle included in each phrase. The verb participle can be "add" or "put". For example, the added verb is "add", and the phrase information obtained is "beef", "add", "two pounds", "shelf life", "3 days"; "pork", "add", "three pounds".
[0224] In some embodiments, step S64 (displaying the semantic understanding content corresponding to at least one of the ingredient segmentations) can be implemented as follows: in response to a preset operation of the display device, displaying the semantic understanding content corresponding to at least one of the ingredient segmentations, or, when displaying the semantic understanding content corresponding to at least one of the ingredient segmentations, hiding the verb segmentations included in the semantic understanding content.
[0225] The preset operation can be closing the refrigerator door, triggering a preset gesture / button, etc. to call up the display interface. In addition, after the display interface is called up in response to the preset operation, the refrigerator will no longer display the interface repeatedly after detecting that it is closed, avoiding multiple displays and improving the user experience.
[0226] For example, in response to a preset operation of the display device, one display mode may be as shown in FIG12C , displaying semantic understanding content corresponding to at least one ingredient segmentation, including the verb segmentation corresponding to the subject. Another display mode may be as shown in FIG12D , displaying semantic understanding content corresponding to at least one ingredient segmentation while hiding the verb segmentation included in the semantic understanding content.
[0227] In addition, when the food stored in the refrigerator changes, parameters such as food type and attribute identification can be modified accordingly. For example, the user stores "two pounds of beef with a shelf life of three days and three pounds of pork" in the morning, takes out one pound of beef and one pound of pork for lunch at noon, and then says to the refrigerator "change the beef to one pound with a shelf life of three days and the pork to two pounds". As shown in Figure 12E, the display device of the refrigerator displays the semantic understanding content corresponding to at least one food segmentation after the modification.
[0228] In an embodiment of the present application, the display device first obtains the word segmentation information corresponding to the voice ingredient data, wherein the word segmentation information includes at least one ingredient segmentation and an attribute identifier corresponding to the ingredient segmentation, and then determines the target segmentation in the at least one ingredient segmentation based on the attribute identifier, and then determines the semantic understanding content corresponding to the at least one ingredient segmentation based on the order relationship between the target segmentation and other segmentations in the at least one ingredient segmentation, and finally displays the semantic understanding content corresponding to the at least one ingredient segmentation. By obtaining at least one ingredient segmentation corresponding to the voice ingredient data and an attribute identifier corresponding to the ingredient segmentation, it is possible to understand multiple ingredients and the attributes of each ingredient and their matching relationships, thereby improving the display device's ability to understand inverted and disordered speech in ingredient management, further accurately displaying it, and improving the user experience.
[0229] In an embodiment of the present application, the display device first obtains the word segmentation information corresponding to the voice ingredient data, wherein the word segmentation information includes at least one ingredient segmentation and an attribute identifier corresponding to the ingredient segmentation, and then determines the target segmentation in the at least one ingredient segmentation based on the attribute identifier, and then determines the semantic understanding content corresponding to the at least one ingredient segmentation based on the order relationship between the target segmentation and other segmentations in the at least one ingredient segmentation, and finally displays the semantic understanding content corresponding to the at least one ingredient segmentation. By obtaining at least one ingredient segmentation corresponding to the voice ingredient data and an attribute identifier corresponding to the ingredient segmentation, it is possible to understand multiple ingredients and the attributes of each ingredient and their matching relationships, thereby improving the display device's ability to understand inverted and disordered speech in ingredient management, further accurately displaying it, and improving the user experience.
[0230] An embodiment of the present application provides a computer device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the semantic understanding methods described in the embodiments of the present application.
[0231] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion of some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. A display device, comprising: a display configured to display an image and / or a user interface; The controller is configured to: receive a user's target control request for the display device; In response to the target control request, a target control text is obtained, and semantic matching is performed on the target control text based on a preset text library to obtain a semantic matching result, wherein the preset text library includes: preset terminology information and preset resource information; Based on the semantic matching result, the control instruction corresponding to the target control request is executed.
2. The display device according to claim 1, wherein the controller is specifically configured to: Performing semantic matching on the target control text based on preset terminology information; It is detected that the target control text does not match any of the terms included in the preset term information, and semantic matching is performed on the target control text based on preset resource information to obtain a semantic matching result.
3. The display device according to claim 2, wherein the controller is specifically configured to: detecting that the target control text matches a candidate term included in the preset term information, and determining that the control instruction corresponding to the target control request is a first control instruction corresponding to the candidate term; Execute a first control instruction corresponding to the candidate term.
4. The display device according to claim 2, wherein the controller is specifically configured to: detecting that the target control text matches a candidate resource included in the preset resource information, and determining that the control instruction corresponding to the target control request is a second control instruction corresponding to the candidate resource; Execute the second control instruction corresponding to the candidate resource.
5. The display device according to claim 1 , wherein the controller is further configured to: In response to the first information sending request, receiving the preset term information sent by the server, wherein: The preset term information is determined by the server based on the user's historical control requests; The preset term information is added to the preset text library.
6. The display device according to claim 1, wherein the controller is further configured to: In response to the second information sending request, receiving the preset resource information sent by the server, wherein: The preset resource information is determined by the server based on the user's historical control requests and the user's historical visits, and the preset resource information includes popular resources in at least two fields; The preset resource information is added to the preset text library.
7. The display device according to claim 2, wherein the controller is further configured to: detecting that the target control text does not match any of the resources included in the preset resource information, sending the target control text to a server so that the server performs semantic analysis on the target control text and determines a third control instruction corresponding to the target control text; In response to a third control instruction corresponding to the target control text sent by the server, the third control instruction is executed.
8. The display device according to claim 1, wherein the controller is further configured to: obtain word segmentation information corresponding to the voice ingredient data, the word segmentation information comprising: At least one food participle and an attribute identifier corresponding to the food participle; Determining a target participle in at least one of the ingredient participles based on the attribute identifier; Based on the sequential relationship between the target segmentation and other segmentations in at least one of the ingredient segmentations, semantic understanding content corresponding to at least one of the ingredient segmentations is determined.
9. The display device according to claim 8, wherein the target participle is the first subject participle that is arranged first in the at least one ingredient participle; The controller is further configured to: determine phrase information based on the attribute identifier of the first traversal segmentation in at least one of the ingredient segmentations, and the sequential relationship between the first traversal segmentation and the target segmentation; and determine the semantic understanding content corresponding to at least one of the ingredient segmentations based on the phrase information.
10. The display device according to claim 9, wherein the controller is further configured to: determine, based on an attribute identifier of a first traversal participle in at least one of the ingredient participles, that the first traversal participle is not a second subject participle, and determine that there is no participle with the same attribute identifier as the first traversal participle; and determine phrase information based on a sequential relationship between the first traversal participle and the target participle.
11. The display device according to claim 10, the controller is further configured to: determine that the sequential relationship between the first traversal participle and the target participle belongs to a preset relationship, and determine that there is an object word between the first traversal participle and the target participle; form the object word and the participles corresponding to the object word into a first phrase, and determine the phrase information based on other participles between the first traversal participle and the target participle and the first phrase.
12. The display device according to claim 11, wherein the controller is further configured to: determine, based on an attribute identifier of a second traversal participle in at least one of the ingredient participles, that the second traversal participle is a second subject participle, and form a second phrase based on other participles between the target participle and the first traversal participle and the first phrase; determine that the second traversal participle is the last second subject participle that has not been traversed, and form a third phrase with the second traversal participle and the remaining participles in at least one of the ingredient participles that have not been traversed; Phrase information is determined based on a sequential relationship between the second phrase and the third phrase.
13. The display device according to claim 12, wherein the controller is further configured to: determine candidate information based on a sequential relationship between the second phrase and the third phrase; and add verb participles to the candidate information to obtain phrase information.
14. The display device according to claim 13, wherein the display is further configured to: display the semantic understanding content corresponding to at least one of the ingredient segmentations in response to a preset operation of the display device, or, when displaying the semantic understanding content corresponding to at least one of the ingredient segmentations, hide the verb segmentations included in the semantic understanding content.
15. A control method for a display device, comprising: receiving a user's target control request for a display device; In response to the target control request, a target control text is obtained, and semantic matching is performed on the target control text based on a preset text library to obtain a semantic matching result, wherein the preset text library includes: preset terminology information and preset resource information; Based on the semantic matching result, executing the control instruction corresponding to the target control request; Display the response interface corresponding to executing the control instruction.
16. The method according to claim 15, further comprising: Obtaining word segmentation information corresponding to the voice ingredient data, the word segmentation information including: at least one ingredient segmentation and an attribute identifier corresponding to the ingredient segmentation; Determining a target participle in at least one of the ingredient participles based on the attribute identifier; Determining semantic understanding content corresponding to at least one of the ingredient segmentations based on a sequential relationship between the target segmentation and other segmentations in at least one of the ingredient segmentations; Display semantic understanding content corresponding to at least one of the ingredient segmentations.
