Human-computer interaction method and device, terminal and storage medium

Abstract

The invention relates to a man-machine interaction method and device, a terminal and a storage medium, and relates to the technical field of terminals. Comprising the steps that a virtual space corresponding to a real space is displayed on a terminal screen, the virtual space is used for containing a virtual object corresponding to equipment, and the virtual object is used for representing virtualized equipment; and in response to a first touch operation executed on the virtual object, controlling the equipment to execute a control instruction indicated by the first touch operation. By using the man-machine interaction method and device, the terminal and the storage medium provided by the invention, the interestingness of man-machine interaction can be improved, and the user experience can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal technology, and in particular to a human-computer interaction method, device, terminal and storage medium. Background Art

[0002] Currently, there are foldable phones (flip) on the market. Foldable phones usually have two screens, an inner screen and an outer screen. The outer screen is usually smaller than the inner screen, and the outer screen is used to display information such as incoming calls, text messages and notifications.

[0003] However, this human-computer interaction method allows users to only use the external screen to view information such as incoming calls, text messages, and notifications, and cannot interact with the external screen in an interesting way. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a human-computer interaction method, device, terminal and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a human-computer interaction method is provided, including:

[0006] Displaying a virtual space corresponding to the real space on a terminal screen, wherein the virtual space is used to accommodate virtual objects corresponding to the device, and the virtual objects are used to represent the virtualized device;

[0007] In response to a first touch operation performed on the virtual object, the device is controlled to execute a control instruction indicated by the first touch operation.

[0008] Optionally, in response to the first touch operation performed on the virtual object, controlling the device to execute a control instruction indicated by the first touch operation includes:

[0009] In response to a first touch operation performed on the virtual object, determining the device corresponding to the virtual object based on a correspondence relationship between the virtual object and the device;

[0010] The device is controlled by the control instruction.

[0011] Optionally, the method further comprises:

[0012] determining whether first device information of the generated virtual object matches second device information of the device stored in the terminal;

[0013] When the first device information matches the second device information, a corresponding relationship between the virtual object and the device is established.

[0014] Optionally, the method further comprises:

[0015] In response to a new operation on the virtual space, obtain virtual space parameters required for creating the virtual space, where the virtual space parameters are determined by user settings or by image information of the real space from multiple angles scanned by the user;

[0016] Create the virtual space according to the virtual space parameters.

[0017] Optionally, the method further includes:

[0018] In response to an operation of creating the virtual object in the virtual space, create the virtual object in the virtual space.

[0019] Optionally, the step of, in response to an operation of creating a virtual object in the virtual space, creating the virtual object in the virtual space includes:

[0020] In response to an operation of creating the virtual object, obtain virtual object parameters required for creating the virtual object, where the virtual object parameters are determined by user settings or by image information of the device from multiple angles scanned by the user;

[0021] Create the virtual object according to the virtual object parameters.

[0022] Optionally, the method further includes:

[0023] Display an application identifier of an application function in the virtual space;

[0024] In response to a second touch operation on the application identifier, start the application function.

[0025] Optionally, displaying a virtual object corresponding to the device on the terminal screen includes:

[0026] Control the virtual object to be displayed in different virtual spaces at different time periods.

[0027] Optionally, the method further includes:

[0028] In response to a third touch operation performed on the virtual object, control at least one of the angle, size, and position of the virtual object to change.

[0029] Optionally, the terminal is a foldable terminal, and the foldable terminal includes an outer screen. Displaying a virtual object corresponding to the device on the terminal screen includes: Displaying the virtual object on the outer screen.

[0030] Optionally, displaying a virtual space corresponding to the real space on the terminal screen includes:

[0031] Determine the terminal position of the terminal;

[0032] When the terminal position is within the position range of the real space corresponding to the virtual space, the virtual space is displayed on the terminal screen.

[0033] Optionally, displaying the virtual space corresponding to the real space on the terminal screen includes:

[0034] determining a network to which the terminal is connected;

[0035] When the network connected to the terminal is within the network range of the real space corresponding to the virtual space, the virtual space is displayed on the terminal screen.

[0036] Optionally, displaying the virtual space corresponding to the real space on the terminal screen includes:

[0037] In response to a selection operation on the virtual space, the selected virtual space is presented on the terminal screen.

[0038] According to a second aspect of an embodiment of the present disclosure, a human-computer interaction device is provided, including:

[0039] A display module is configured to display a virtual space corresponding to the real space on a terminal screen, wherein the virtual space is used to accommodate virtual objects corresponding to the device, and the virtual objects are used to represent the virtualized device;

[0040] The control module is configured to control the device to execute a control instruction indicated by a first touch operation in response to the first touch operation performed on the virtual object.

[0041] According to a third aspect of an embodiment of the present disclosure, a terminal is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to: execute the steps of the human-computer interaction method provided in the first aspect of the present disclosure.

[0042] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the human-computer interaction method provided in the first aspect of the present disclosure are implemented.

[0043] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0044] On the one hand, virtual objects of devices that appear in real life can be displayed on the terminal screen, thereby enhancing the authenticity of the displayed virtual objects; on the other hand, the working status of the devices in real life can be controlled through the first touch operation performed by the user on the virtual objects, so that the user can control the devices in real life by manipulating the virtual objects displayed on the terminal screen, thereby improving the human-computer interaction experience and allowing users to remotely control devices in a more interesting way.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0047] Figure 1 The present invention is a flowchart of a human-computer interaction method according to an exemplary embodiment.

[0048] Figure 2 The figure is a schematic diagram of a virtual object of a touch-controlled ceiling lamp according to an exemplary embodiment.

[0049] Figure 3 The diagram is a schematic diagram of an adjustment interface that jumps after touching a ceiling lamp according to an exemplary embodiment.

[0050] Figure 4 The figure is a schematic diagram of a virtual object of a touch-controlled air conditioner according to an exemplary embodiment.

[0051] Figure 5 It is a schematic diagram of an interface for setting virtual space parameters according to an exemplary embodiment.

[0052] Figure 6 The figure is a schematic diagram of an interface for adding multiple pictures at different angles according to an exemplary embodiment.

[0053] Figure 7 It is a schematic diagram showing a method of selecting different virtual spaces according to an exemplary embodiment.

[0054] Figure 8 The figure is a schematic diagram showing application identifiers of various application functions according to an exemplary embodiment.

[0055] Fig. 9 The diagram is a schematic diagram showing displaying an application logo in a virtual space according to an exemplary embodiment.

[0056] Fig.10 It is a schematic diagram showing a human-computer interaction device according to an exemplary embodiment.

[0057] Fig.11 It is a schematic diagram of a terminal according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0059] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0060] Figure 1 is a flow chart of a human-computer interaction method according to an exemplary embodiment. Figure 1 As shown, the human-computer interaction method is used in a terminal and includes the following steps.

[0061] In step S11, a virtual space corresponding to the real space is displayed on the terminal screen, and the virtual space contains virtual objects.

[0062] Virtual space is a virtualized mapping of real space, and is used to accommodate virtual objects corresponding to devices. For example, when a user walks into the office area, the virtual space displayed on the terminal screen is the virtualized space of the office area, and when a user walks into the bedroom, the virtual space displayed on the terminal screen is the virtualized space of the bedroom.

[0063] Device refers to a device that has established a connection with a terminal. The connection includes wired connection and wireless connection. Wireless connection includes Bluetooth connection and WIFI connection, etc. Wired connection includes the connection between the device and the terminal established by optical fiber. Devices can be artificial intelligence devices (AI) or Internet of Things devices (IOT). IoT devices include cameras, air conditioners, washing machines, desk lamps and other devices. Artificial intelligence devices include smart devices such as speakers, robots and robot dogs.

[0064] Virtual objects are used to represent virtualized devices. Virtual objects are visualized 3D models of devices. For example, the terminal screen displays 3D models of virtual cameras, air conditioners, washing machines, and other devices. In real life, there are also real cameras, air conditioners, washing machines, and other devices.

[0065] The terminal may be a foldable mobile phone, which includes an inner screen and an outer screen, and the back screen of the outer screen is referred to as the back screen, and the virtual object may be displayed on the back screen. The terminal screen referred to in the present disclosure is not limited to the back screen of a foldable mobile phone, but may also be applied to the inner screen of a foldable mobile phone, and of course may also be applied to the screen of a non-foldable mobile phone.

[0066] The terminal screen can be kept always on, so that when the user sees the terminal screen, the user can directly see the virtual object displayed on the terminal screen; the virtual object can also be displayed on the terminal screen when the terminal screen is awakened by the user.

[0067] In step S12, in response to a first touch operation performed on the virtual object, the device is controlled to execute a control instruction indicated by the first touch operation.

[0068] The first touch operation is used to control the real device to perform a corresponding action after touching the virtual object. The first touch operation includes a first operation and a second operation.

[0069] The first operation is used to touch the virtual object to open the adjustment interface. The first operation can be either double-clicking the virtual object or long pressing the virtual object. The adjustment interface is used to display the control buttons for manipulating the virtual object. If the device is a desk lamp, the adjustment interface can display the brightness adjustment bar and the on / off switch of the desk lamp, etc.; if the device is an air conditioner, the adjustment interface can display the temperature adjustment button and the on / off switch of the air conditioner, etc.

[0070] The second operation is to touch the control button of the adjustment interface to control the device to perform corresponding actions, including the on / off control of the device and the increase / decrease control of the device parameters. If the device is a desk lamp, the second operation can be to adjust the brightness of the desk lamp on the adjustment interface; if the device is an air conditioner, the second operation can be to adjust the temperature of the air conditioner on the adjustment interface.

[0071] In order to ensure the user experience, when the user touches the virtual object, in addition to controlling the real device to perform the corresponding action, the display status of the virtual object will also be displayed on the terminal screen, so that the user can know the working status of the real device through the display status of the virtual object.

[0072] For example, see Figure 2 As shown in the figure, the terminal screen displays the ceiling light. When the user double-clicks the ceiling light, a pop-up window will appear on the terminal screen. Figure 3 The adjustment interface shown in the figure has a ceiling light on / off switch and a light adjustment bar. When the user turns the ceiling light on / off switch to on, the ceiling light on the terminal screen is in a lit display state, and at the same time, the real ceiling light connected to the terminal is also in a lit working state. Then, when the user sees that the ceiling light on the terminal screen is lit, he or she can also know that the real ceiling light is also lit.

[0073] For example, see Figure 4 As shown, the air conditioner is displayed on the terminal screen. When the user double-clicks the air conditioner, an adjustment interface will pop up on the terminal screen. The adjustment interface has an air conditioner on / off switch and a temperature adjustment button. When the user turns the air conditioner on / off switch to on, the air conditioner on the terminal screen is in an on display state, and the real air conditioner connected to the terminal is also in an on working state. So when the user sees the air conditioner on the terminal screen in an on display state, he or she can also know that the real air conditioner is also turned on.

[0074] Through the above technical solution, on the one hand, virtual objects of devices that appear in real life can be displayed on the terminal screen, thereby increasing the interest of human-computer interaction; on the other hand, the working state of the device in real life can be controlled through the first touch operation performed by the user on the virtual object, so that the user can control the device in real life by manipulating the virtual object displayed on the terminal screen, further improving the human-computer interaction experience and allowing users to remotely control the device in a more interesting way.

[0075] The following introduces a specific embodiment involved in the above step S11, which is used to explain the specific steps of establishing a virtual space, where the virtual space is a space used to accommodate virtual objects.

[0076] In step S21, in response to the creation operation of the virtual space, virtual space parameters required for creating the virtual space are acquired.

[0077] The virtual space parameters include the size, color and type of the virtual space. The size includes the length, width and height of the virtual space. The type of the virtual space will change with the changes in different occasions. For example, the types of virtual space include study, bedroom, living room, toilet and office. The virtual space parameters are determined by user settings or by multi-angle image information of the real space scanned by the user. When the user creates a new virtual space, the terminal will obtain the virtual space parameters set by the user when creating the new virtual space, or the terminal will obtain multi-angle image information taken by the user and obtain the virtual space parameters based on the multi-angle image information.

[0078] For example, if the virtual space parameters are set by the user, see Figure 5 As shown, when a user creates a new virtual space, he or she will also set virtual space parameters such as size, color, and type of the virtual space, and the terminal will obtain the virtual space parameters set by the user.

[0079] For example, when the virtual space parameters are obtained based on multi-angle image information, please refer to Figure 6As shown, when a user creates a new virtual space, the terminal will display a photo add button, prompting the user to upload multi-angle pictures of the real space. After the user shoots the real space and uploads multi-angle image information of the real space, the terminal will determine the virtual space parameters required to construct the virtual space based on the multi-angle image information.

[0080] In step S22, the virtual space is established according to the virtual space parameters.

[0081] After obtaining the virtual space parameters, such as the size, color and type of the virtual space, a virtual space can be fitted according to the size, color and type of the virtual space.

[0082] Of course, when the virtual space is established according to the virtual space parameters, the system will use different space templates to pre-generate different virtual objects in the virtual space. For example, for the virtual space of the living room, the virtual space generated according to the space template of the living room will be accompanied by the display of virtual objects such as carpets, curtain walls, windows, etc., which do not require user settings. These virtual objects are generated along with the generation of the virtual space and are used to distinguish different types of virtual spaces such as study, bedroom, living room, toilet and office, and do not require user settings.

[0083] In step S23, a virtual space corresponding to the real space is displayed on the terminal screen, and the virtual space contains the virtual object.

[0084] After the virtual space is established, the virtual space corresponding to the real space can be displayed on the terminal screen. For example, if there is a real-life study, bedroom, living room, toilet and office in the real space, the virtual space such as the study, bedroom, living room, toilet and office can be displayed on the terminal screen according to the user's settings or the multi-angle image information of the real space taken by the user.

[0085] After the virtual space is established, the user can also control at least one of the angle and size of the virtual space to change in response to the fourth touch operation performed on the virtual space by the user. The user can arbitrarily change the display angle of the displayed virtual space and the size of the virtual space. The fourth touch operation includes a rotation operation and a zoom operation on the virtual space.

[0086] It is understandable that after the virtual space is established through steps S21 and S22, the subsequent step S23 only needs to use the established virtual space for presentation, without having to execute steps S21 and S22 to establish the virtual space every time the virtual space is presented.

[0087] Through the above technical solution, a virtual space can be displayed on the terminal screen, and virtual objects can be displayed in the virtual space, so that users can see various virtual objects in the virtual space, which can be closer to the placement relationship between real-life devices and real houses, making the display of virtual objects more realistic, further enhancing the user's interactive experience.

[0088] The following describes an optional embodiment of step S11 and step S12, which is used to explain how to create a virtual object in a virtual space. After the virtual object is created in the virtual space, the virtual object is bound to the virtual space, and when the virtual space is subsequently displayed, the bound virtual object will also be displayed synchronously in the virtual space.

[0089] Since virtual objects are divided into virtual objects with templates pre-stored in the terminal and virtual objects not pre-stored in the terminal, virtual objects will be established in two cases below.

[0090] For templated virtual objects pre-stored in the terminal, when the user creates a new virtual space, virtual objects related to the virtual space pre-stored in the terminal will be displayed on one side of the terminal screen, and the user can select the virtual objects and drag them into the virtual space to create virtual objects in the virtual space. Different types of virtual spaces are pre-associated with different virtual objects.

[0091] For example, for the virtual space of the office, the terminal pre-associates the relationship between the virtual space of the office and virtual objects such as green plants, office computers, projectors, and projection screens. When a user creates a new virtual space "office", locally stored virtual objects such as office computers, projectors, and projection screens will be displayed on one side of the terminal screen. Users can select the virtual objects and drag them into the virtual space to complete the creation of virtual objects in the virtual space.

[0092] For virtual objects that are not pre-stored in the terminal, the virtual object can be established in the virtual space in response to an operation of establishing the virtual object in the virtual space, including: in response to the operation of establishing the virtual object, obtaining virtual object parameters required for establishing the virtual object, the virtual object parameters are determined by user settings or by image information of a multi-angle device scanned by the user; and establishing the virtual object according to the virtual object parameters.

[0093] The virtual object parameters include the shape, color, and type of the virtual object. The shapes of the virtual objects include: cuboid, irregular shape, cylinder, etc. One device corresponds to one virtual object type, for example, a desk lamp is one type, and a desk is another type.

[0094] In the case where the virtual space parameters are set by the user, when the user creates a new virtual object, the user can set the virtual object parameters such as the shape, color and type of the virtual object, and the terminal creates the virtual object according to the virtual object parameters. For example, when the user creates a new "desk" virtual object, the user can set the desk to be rectangular and the color of the desk to be pink, then the terminal will generate the desk according to the two virtual object parameters of pink and rectangle.

[0095] In the case where the virtual space parameters are obtained based on multi-angle image information, when the user creates a new virtual space, the terminal will prompt the user to upload multi-angle pictures of the device. After the user takes a photo of the real device and uploads multi-angle image information of the device, the terminal will determine the virtual object parameters based on the multi-angle image information and fit the virtual object based on the virtual object parameters. Of course, if the user uploads a single device photo, the terminal can display the photo as a virtual object in the virtual space.

[0096] For any favorite item in life, the user can take a photo of the item. The terminal generates a three-dimensional virtual object of the item based on the multi-angle pictures of the item, and then displays the virtual object in the virtual space.

[0097] After the virtual object is created, at least one of the angle, size and position of the virtual object can be controlled to change in response to a third touch operation performed on the virtual object. The third touch operation includes at least one operation such as a rotation operation, a zoom operation and a drag operation.

[0098] For example, the user creates a desk lamp and a desk in the virtual space, and the desk lamp is located on the left side of the desk. If the user wants to change the position of the desk lamp, the user can perform a third touch operation on the desk lamp and drag the desk lamp to any position in the virtual space.

[0099] For some dynamic virtual objects, such as virtual objects such as pet dogs and pet cats created by users, these dynamic virtual objects can be controlled to be displayed in different virtual spaces at different time periods.

[0100] For example, after the user creates a virtual object of a pet cat, the user can control the pet cat to appear in the virtual space of the "bedroom" during the night time period, and control the pet cat to appear in the virtual space of the "office" during the daytime.

[0101] Through the above technical solution, on the one hand, the terminal can create virtual objects that the user wants to see in the virtual space according to the user's preferences, and after the user creates the virtual object, the angle, size and position of the virtual object can be changed arbitrarily, so that the appearance and position of the virtual object are in line with the user's preferences, further enhancing the sense of human-computer interaction; on the other hand, since the dynamic virtual objects created by the user will randomly appear in different virtual spaces at different time periods, the fun of the virtual space is also increased.

[0102] The following introduces an optional embodiment involved in step S11 and step S12, which is used to explain how to associate a virtual object with a device in a terminal so as to control the device while regulating the virtual object.

[0103] Since the terminal stores information about real devices that have established connections, but does not store information about real devices that have not established connections, the following will explain examples of controlling virtual objects in two situations: storing device information and not pre-saving device information.

[0104] For devices stored in the terminal, for example, the terminal has established communication connections with cameras, speakers, washing machines, air conditioners and other devices, so the terminal will store the device information of these devices, which includes the appearance and model of the device. When the user creates a new virtual object in the virtual space, the terminal will automatically determine whether the first device information of the generated virtual object matches the second device information of the device stored in the terminal; when the first device information matches the second device information, a corresponding relationship between the virtual object and the device is established. When the user performs a first touch operation on the virtual object, the device corresponding to the virtual object is determined according to the corresponding relationship, so that the device is controlled by the control instruction indicated by the first touch operation.

[0105] For example, the terminal has established a communication connection with the speaker, so the terminal knows the device information of the speaker. When the user creates a virtual object of the speaker in the virtual space, the terminal will determine whether the speaker is the same as the pre-stored speaker. If they are the same, it will output a prompt "The same speaker exists in the system. Do you want to establish a corresponding relationship between the currently established virtual object and the speaker?" If the user clicks "Agree", the terminal will establish a corresponding relationship between the virtual object and the speaker. When the user controls the virtual object to open and control the virtual object to play music, the speaker in real life will be started and automatically play music.

[0106] The first device information includes the shape and model of the virtual object, and the second device information includes the shape and model of the virtual object. Whether the first device information matches the device information includes at least one of the following: the shape of the virtual object is the same as the shape of the device, and the model of the virtual object is the same as the model of the device.

[0107] The second device information stored in the terminal refers to the device information of the device that has established a communication connection with the terminal. For example, if the speaker has been connected to the terminal via a WIFI network, the second device information of the speaker will be stored in the terminal; if the headset has been connected to the terminal via Bluetooth, the second device information of Bluetooth will also be stored in the terminal.

[0108] For devices that are not stored in the terminal, such as desks, benches, photo frames, etc. that cannot establish a communication connection with the terminal, after the user establishes a virtual space, the virtual objects of these devices can be directly established in the virtual space without establishing a corresponding relationship between these devices and the virtual objects. For another example, for devices such as televisions that do not establish a communication connection with the terminal, although the user has established a virtual object of the television in the virtual space, since the terminal has not established a communication connection with the television, after the user performs the first touch operation on the television, the terminal cannot send a control instruction to the television. At this time, the user can be reminded to establish a communication connection between the television and the terminal.

[0109] Through the above technical solution, for devices that have established a communication connection with the terminal, a correspondence between the virtual object and the device can be established. When the user manipulates the virtual object, the control instruction of the first touch operation indicated by the user can be sent to the device according to the correspondence, and the device can perform the corresponding action through the control instruction of the first touch operation indication.

[0110] The following describes an optional embodiment of the above step S11 and step S12, which is used to explain the switching process of the virtual space. The switching process of the virtual space can be automatic or manual, and this embodiment will embody the switching process in different solutions.

[0111] Solution A: determining the terminal position of the terminal; and displaying the virtual space on the terminal screen when the terminal position is within the position range of the real space corresponding to the virtual space.

[0112] When creating a virtual space, the user can establish a geographical location range of the real space corresponding to the virtual space, so that when the current terminal position is within the geographical location range of the real space corresponding to the virtual space, the virtual space can be displayed on the terminal screen.

[0113] The terminal location refers to the GPS location of the terminal, and the location range in real space is an area range.

[0114] For example, the user establishes a geographical location range of the real space "bedroom" corresponding to virtual space A, and also establishes a geographical location range of the real space "office" corresponding to virtual space B. When the user's handheld terminal enters the geographical location range of "bedroom", the virtual space A corresponding to "bedroom" will be automatically displayed on the terminal screen; when the user's handheld terminal enters the geographical location range of "office", the virtual space B corresponding to "office" will be automatically displayed on the terminal screen.

[0115] Solution B: Determine the network to which the terminal is connected; when the network to which the terminal is connected is within the network range of the real space corresponding to the virtual space, display the virtual space on the terminal screen.

[0116] When establishing a virtual space, the user may also establish a network range of a real space corresponding to the virtual space, so that when the network currently connected to the terminal is within the network range of the real space corresponding to the virtual space, the virtual space may be displayed on the terminal screen.

[0117] The network to which the terminal is connected may be a local area network, and the network range of the real space is a local area network range.

[0118] For example, the user has established a local area network "Hahaha" for the real space "bedroom" corresponding to virtual space A, and has also established a local area network "Hehehe" for the real space "office" corresponding to virtual space B. When the user holds a terminal and enters the "bedroom" and connects to the local area network "Hahaha", the virtual space A corresponding to the "bedroom" will be automatically displayed on the terminal screen; when the user holds a terminal and enters the "office" and connects to the local area network "Hehehe", the virtual space B corresponding to the "office" will be automatically displayed on the terminal screen.

[0119] Solution C: In response to a selection operation on the virtual space, the selected virtual space is displayed on the terminal screen.

[0120] See also Figure 7 As shown, there is a selection bar for back space selection on the terminal screen, and when the user slides the selection bar, he can switch between different virtual spaces.

[0121] Through Scheme A and Scheme B in the above technical scheme, the current location of the terminal and the network to which the user terminal is currently connected can be automatically identified, and based on the current location of the terminal or the network to which the terminal is currently connected, after the user reaches a different real space, it is preferentially switched to the virtual space corresponding to the real space without the need for the user to switch, and the virtual space displayed on the terminal screen can also change with the change of the user's location, so that the virtual space displayed on the terminal screen can be closer to real life; through Scheme C in the above technical scheme, when the virtual space currently displayed on the terminal screen is not the virtual space the user wants, the user can automatically manually select a different virtual space on the terminal screen to better meet the user's preferences and improve the user experience.

[0122] The following introduces an optional embodiment involved in the above-mentioned step S11 and step S12, which is used to explain how to coordinate the application identifier of the application function originally displayed on the terminal screen with the virtual space proposed in the present disclosure.

[0123] See also Figure 8 As shown, the outer screen of a foldable phone is usually used to display the following application functions:

[0124] The first application function displays information such as incoming calls, text messages, and notifications: When there is an incoming call, text message, or other notification, the external screen will display the relevant information, and the user can quickly view it through the external screen and decide whether to answer or reply.

[0125] Second application function, taking photos and selfies: The external screen of some folding mobile phones can be used as a selfie camera, and users can take their own photos or videos through the external screen.

[0126] The second application function is to control music playback: users can control music playback through the external screen, such as switching songs, adjusting volume, etc.

[0127] The fourth application function is to display time, weather and other information: the external screen can display the current time, date, weather and other information, so that users can check it at any time.

[0128] Fifth application function: The external screen of some foldable mobile phones can also be used to display sports data, control alarms, view calendars, etc. The specific functions depend on the design of the mobile phone and software support.

[0129] In order to coordinate the display of the original application functions and the virtual space proposed in this disclosure, it can be achieved in the following two ways:

[0130] In a first manner, an application identifier of an application function is displayed in the virtual space; and in response to a second touch operation on the application identifier, the application function is started.

[0131] See also Fig. 9As shown, an application identifier of an application function can be established in the virtual space, and the user can trigger a second touch operation on the application identifier to start the application function and display a page linked to the application function on the external screen; after closing the application function, the user returns to the display interface of the virtual space.

[0132] In the second mode, there is an exit button on the terminal screen. In response to the user clicking the exit button, the terminal exits the interface displaying the virtual space and returns to the interface displaying the application logos of multiple application functions.

[0133] It can be understood that the solution proposed in the embodiment of the present disclosure can be applied to the terminal screen of a non-folding mobile phone, and can also be applied to the external screen of a folding mobile phone or the internal screen of a folding mobile phone, and the present disclosure does not impose any restrictions on this.

[0134] Through the first method in the above technical solution, the application logo can be displayed in the virtual space, and the application function can be started after the user triggers the application logo, so that the user can also use the application function normally on the display page of the virtual space; through the second method, after exiting the interface of the virtual space, it is possible to return to the interface where the application logo is displayed, so as to avoid affecting the user's use of normal application functions, and after the virtual space no longer carries the application logo, there is more space to accommodate the remaining virtual objects.

[0135] Fig.10 is a block diagram of a human-computer interaction device according to an exemplary embodiment. Fig.10 The human-computer interaction device 1000 includes: a display module 1020 and a control module 1030 .

[0136] A display module 1020 is configured to display a virtual space corresponding to a real space on a terminal screen, wherein the virtual space is used to accommodate virtual objects corresponding to the device, and the virtual objects are used to represent the virtualized device;

[0137] The control module 1030 is configured to control the device to execute a control instruction indicated by the first touch operation in response to the first touch operation performed on the virtual object.

[0138] Optionally, the control module 1030 includes:

[0139] A first response submodule is configured to determine the device corresponding to the virtual object based on the correspondence relationship between the virtual object and the device in response to a first touch operation performed on the virtual object;

[0140] The control submodule is configured to control the device according to the control instruction.

[0141] Optionally, the human-computer interaction device 1000 includes:

[0142] a matching module configured to determine whether the first device information of the generated virtual object matches the second device information of the device stored in the terminal;

[0143] The correspondence establishing module is configured to establish a correspondence between the virtual object and the device when the first device information matches the second device information.

[0144] Optionally, the human-computer interaction device 1000 includes:

[0145] A first response module is configured to obtain virtual space parameters required for creating the virtual space in response to a creation operation of the virtual space, wherein the virtual space parameters are determined by user settings or by image information of a real space from multiple angles scanned by the user;

[0146] The virtual space establishment module is configured to establish the virtual space according to the virtual space parameters.

[0147] Optionally, the human-computer interaction device 1000 includes:

[0148] The virtual object creation module is configured to create the virtual object in the virtual space in response to an operation of creating the virtual object in the virtual space.

[0149] Optionally, the human-computer interaction device 1000 includes:

[0150] an application identification display module, configured to display an application identification of an application function in the virtual space;

[0151] The second response module is configured to start the application function in response to a second touch operation on the application identifier.

[0152] Optionally, the display module 1020 includes:

[0153] The fourth display submodule is configured to control the virtual object to be displayed in different virtual spaces in different time periods.

[0154] Optionally, the human-computer interaction device 1000 includes:

[0155] The third response module is configured to control at least one of the angle, size and position of the virtual object to change in response to a third touch operation performed on the virtual object.

[0156] Optionally, the terminal is a foldable terminal, the foldable terminal includes an external screen, and the display module 1020 includes:

[0157] The fourth display submodule is configured to display the virtual object on the external screen.

[0158] Optionally, the virtual object creation module includes:

[0159] A second response submodule is configured to, in response to the operation of establishing the virtual object, obtain virtual object parameters required for establishing the virtual object, wherein the virtual object parameters are determined by user settings or by image information of a multi-angle device scanned by the user;

[0160] The virtual object establishment submodule is configured to establish the virtual object according to the virtual object parameters.

[0161] Optionally, the display submodule includes:

[0162] A terminal location determination submodule, configured to determine the terminal location of the terminal;

[0163] The second display submodule is configured to display the virtual space on the terminal screen when the terminal is located within the position range of the real space corresponding to the virtual space.

[0164] Optionally, the display submodule includes:

[0165] A terminal network determination module, configured to determine the network to which the terminal is connected;

[0166] The third display submodule is configured to display the virtual space on the terminal screen when the network to which the terminal is connected is within the network range of the real space corresponding to the virtual space.

[0167] Optionally, the display submodule includes:

[0168] The third response submodule is configured to display the selected virtual space on the terminal screen in response to the selection operation of the virtual space.

[0169] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0170] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the human-computer interaction method provided by the present disclosure are implemented.

[0171] Fig.111 is a block diagram of a device 1100 for human-computer interaction according to an exemplary embodiment. For example, the device 1100 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or other terminals.

[0172] Reference Fig.11 , the device 1100 may include one or more of the following components: a processing component 1102 , a memory 1104 , a power component 1106 , a multimedia component 1108 , an audio component 1110 , an input / output interface 1112 , a sensor component 1114 , and a communication component 1116 .

[0173] The processing component 1102 generally controls the overall operation of the device 1100, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to complete all or part of the steps of the above-mentioned human-computer interaction method. In addition, the processing component 1102 may include one or more modules to facilitate the interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate the interaction between the multimedia component 1108 and the processing component 1102.

[0174] The memory 1104 is configured to store various types of data to support the operation of the device 1100. Examples of such data include instructions for any application or method operating on the device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0175] The power supply component 1106 provides power to the various components of the device 1100. The power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 1100.

[0176] The multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front camera and / or a rear camera. When the device 1100 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0177] The audio component 1110 is configured to output and / or input audio signals. For example, the audio component 1110 includes a microphone (MIC), and when the device 1100 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1104 or sent via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting audio signals.

[0178] The input / output interface 1112 provides an interface between the processing component 1102 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0179] The sensor assembly 1114 includes one or more sensors for providing various aspects of the status assessment of the device 1100. For example, the sensor assembly 1114 can detect the open / closed state of the device 1100, the relative positioning of components, such as the display and keypad of the device 1100, the sensor assembly 1114 can also detect the position change of the device 1100 or a component of the device 1100, the presence or absence of user contact with the device 1100, the orientation or acceleration / deceleration of the device 1100, and the temperature change of the device 1100. The sensor assembly 1114 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1114 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0180] The communication component 1116 is configured to facilitate wired or wireless communication between the device 1100 and other devices. The device 1100 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0181] In an exemplary embodiment, the device 1100 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned human-computer interaction method.

[0182] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, and the instructions can be executed by a processor 1120 of the device 1100 to complete the above-mentioned human-computer interaction method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0183] In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be an IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC, system on chip or system-level chip), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned human-computer interaction method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, such as the integrated circuit or chip including a processor, a memory, and an interface for communicating with other devices. The executable instruction can be stored in the memory, and when the executable instruction is executed by the processor, the above-mentioned human-computer interaction method is implemented; alternatively, the integrated circuit or chip can receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned human-computer interaction method.

[0184] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above human-computer interaction method when executed by the programmable device.

[0185] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0186] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A human-computer interaction method, It is characterized in that include: Displaying a virtual space corresponding to the real space on a terminal screen, wherein the virtual space is used to accommodate virtual objects corresponding to the device, and the virtual objects are used to represent the virtualized device; In response to a first touch operation performed on the virtual object, the device is controlled to execute a control instruction indicated by the first touch operation.

2. The method according to claim 1, It is characterized in that The step of controlling the device to execute a control instruction indicated by the first touch operation in response to the first touch operation performed on the virtual object includes: In response to a first touch operation performed on the virtual object, determining the device corresponding to the virtual object based on a correspondence relationship between the virtual object and the device; The device is controlled by the control instruction.

3. The method according to claim 2, It is characterized in that The method further comprises: determining whether first device information of the generated virtual object matches second device information of the device stored in the terminal; When the first device information matches the second device information, a corresponding relationship between the virtual object and the device is established.

4. The method according to claim 1, It is characterized in that The method further comprises: In response to a new creation operation of the virtual space, obtaining virtual space parameters required for creating the virtual space, wherein the virtual space parameters are determined by user settings or by image information of a real space from multiple angles scanned by the user; The virtual space is established according to the virtual space parameters.

5. The method according to claim 1, It is characterized in that The method further comprises: In response to an operation of establishing the virtual object in the virtual space, the virtual object is established in the virtual space.

6. The method according to claim 5, It is characterized in that In response to the operation of establishing the virtual object in the virtual space, establishing the virtual object in the virtual space includes: In response to the operation of establishing the virtual object, obtaining virtual object parameters required for establishing the virtual object, wherein the virtual object parameters are set by a user or determined by image information of a multi-angle device scanned by a user; The virtual object is created according to the virtual object parameters.

7. The method according to claim 1, It is characterized in that The method further comprises: An application identifier for displaying application functions in the virtual space; In response to a second touch operation on the application identifier, the application function is started.

8. The method according to claim 1, It is characterized in that Display virtual objects corresponding to the device on the terminal screen, including: The virtual object is controlled to be displayed in different virtual spaces at different time periods.

9. The method according to claim 1, It is characterized in that The method further comprises: In response to a third touch operation performed on the virtual object, at least one of an angle, a size, and a position of the virtual object is controlled to change.

10. The method according to claim 1, It is characterized in that The terminal is a foldable terminal, the foldable terminal includes an external screen, and displaying a virtual object corresponding to the device on the terminal screen includes: The virtual object is displayed on the external screen.

11. The method according to any one of claims 1 to 10, It is characterized in that The displaying of the virtual space corresponding to the real space on the terminal screen includes: determining a terminal position of the terminal; When the terminal position is within the position range of the real space corresponding to the virtual space, the virtual space is displayed on the terminal screen.

12. The method according to any one of claims 1 to 10, It is characterized in that The displaying of the virtual space corresponding to the real space on the terminal screen includes: determining a network to which the terminal is connected; When the network connected to the terminal is within the network range of the real space corresponding to the virtual space, the virtual space is displayed on the terminal screen.

13. The method according to any one of claims 1 to 10, It is characterized in that The displaying of the virtual space corresponding to the real space on the terminal screen includes: In response to a selection operation on the virtual space, the selected virtual space is presented on the terminal screen.

14. A human-computer interaction device, It is characterized in that include: A display module is configured to display a virtual space corresponding to the real space on a terminal screen, wherein the virtual space is used to accommodate virtual objects corresponding to the device, and the virtual objects are used to represent the virtualized device; The control module is configured to control the device to execute a control instruction indicated by a first touch operation in response to the first touch operation performed on the virtual object.

15. A terminal, It is characterized in that include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Execute the steps of any one of the methods of claims 1 to 13.

16. A computer readable storage medium having computer program instructions stored thereon, It is characterized in that When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 13 are implemented.

Images