Story scenario-based human-computer interaction method, device, equipment and medium

Abstract

The application discloses a story scene-based human-computer interaction method and device, equipment and medium, which are applied to the field of augmented reality. The method is executed by a terminal with a camera. The method comprises the following steps: acquiring a real scene video stream collected by the camera, displaying an AR video stream based on the real scene video stream, and completing an inference task corresponding to the story scene based on the AR video stream. Since the AR background region is obtained by processing the background region in the real scene video stream, and the AR character region is obtained by processing and replacing the foreground character region in the real scene video stream, the visual effect with high immersion can be provided without the help of clothes, props and paper scripts of real objects, and better visual experience can be achieved with reduced consumption of physical resources.

Description

Technical Field

[0001] This application relates to the field of augmented reality (AR), and in particular to a human-computer interaction method, apparatus, device, and medium based on a story scene. Background Technology

[0002] Murder mystery games involve multiple players taking on different roles in a story, with each role following a specific deduction process.

[0003] The offline murder mystery game technology provided by related technologies uses a projector to project images onto the walls of the murder mystery room. These projected images simulate specific murder mystery scenarios, such as a period wedding scene or a Republican-era spy drama scene. Players need to change into designated costumes and props and use paper scripts to complete deductions within the murder mystery room.

[0004] While the aforementioned offline murder mystery games already offer a good immersive experience, achieving this requires significant physical resources, such as costumes, props, and paper scripts specifically designed for the murder mystery scene. This consumption of specially customized physical resources becomes even more pronounced when there are a large number of scripts available for offline murder mystery games. Summary of the Invention

[0005] This application provides a human-computer interaction method, device, equipment, and medium based on a story scenario, which can achieve a superior visual experience while reducing the consumption of physical resources. The technical solution is as follows:

[0006] According to one aspect of this application, a human-computer interaction method based on a story scenario is provided, the method comprising:

[0007] The real-scene video stream captured by the camera is obtained. The video frame of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person.

[0008] The AR video stream is displayed based on the real-world video stream. The video screen of the AR video stream includes an AR background area and an AR character area. The AR background area displays the scene screen of the story scene, and the AR character area displays the real-world character wearing AR costumes. The AR costumes correspond to the story characters in the story scene.

[0009] Complete the reasoning task corresponding to the story scene based on the AR video stream;

[0010] The AR background area is obtained by processing the image content in the background area, and the AR character area is obtained by processing the image content in the foreground character area.

[0011] According to one aspect of this application, a human-computer interaction method based on a story scenario is provided, the method comprising:

[0012] Receive the real-time video stream reported by the receiving terminal;

[0013] Image semantic recognition is performed on the real-scene video frames in the real-scene video stream to obtain the background region and foreground person region in the real-scene video frame; the foreground person region corresponds to the real-scene person.

[0014] The image content in the background area is processed to obtain an augmented reality (AR) background area, and the image content in the foreground character area is processed to obtain an AR character area; the AR background area displays the scene of the story scene, and the AR character area displays the real-world character wearing AR costumes, the AR costumes corresponding to the story characters in the story scene;

[0015] An AR video stream is obtained based on the AR video frames after merging the AR background area and the AR character area;

[0016] The AR video stream is sent to the terminal so that the terminal can complete the reasoning task corresponding to the story scene based on the AR video stream.

[0017] According to one aspect of this application, a human-computer interaction device based on a story scenario is provided, the device comprising:

[0018] The acquisition module is used to acquire the real-scene video stream captured by the camera. The video frame of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person.

[0019] The display module is used to display an AR video stream based on the real-world video stream. The video screen of the AR video stream includes an AR background area and an AR character area. The AR background area displays the scene screen of the story scene, and the AR character area displays the real-world character wearing AR costumes. The AR costumes correspond to the story characters in the story scene.

[0020] The interactive module is used to complete the reasoning task corresponding to the story scene based on the AR video stream;

[0021] The AR background area is obtained by processing the image content in the background area, and the AR character area is obtained by processing the image content in the foreground character area.

[0022] In an optional design, the display module is used to display the AR video stream when the real-world character in the real-world video stream is bound to a first story character in the story scene; the AR character area in the AR video stream displays the real-world character wearing the first AR costume;

[0023] The first AR outfit includes the clothing corresponding to the first story character in the story scene.

[0024] In an optional design, the device further includes a processing module:

[0025] The processing module is used to perform image semantic recognition on video frames in the real-scene video stream to obtain the background region and the foreground person region, wherein the foreground person region includes a face region and a non-face region.

[0026] Facial recognition is performed on the facial area to obtain the identity of the real-world person.

[0027] Based on the identity of the real-life character, determine the first story role that the real-life character is bound to in the story scene;

[0028] The background area is replaced or blended based on the scene materials of the story scene to obtain the AR background area; and the non-face area is replaced or blended based on the character materials of the first story character to obtain the AR character area;

[0029] The display module is used to display the AR video stream based on the AR background area and the AR character area.

[0030] In an optional design, the display module is used to display a second face image of the real-world person not wearing the AR terminal in the second face area of ​​the AR person area.

[0031] In an optional design, the processing module is used to input the first facial image and the sample facial data of the real-world person into a generative network for image reconstruction to obtain a second facial image of the real-world person who is not wearing the AR terminal.

[0032] The display module is used to display the second face image in the second face area of ​​the AR character area.

[0033] In an optional design, the display module is used to complete at least one of the following tasks corresponding to the story scene: information acquisition, evidence collection, and puzzle reasoning, based on the AR information displayed in the AR video stream.

[0034] In an optional design, the interactive module is used to obtain character information of the first story character;

[0035] The display module is used to display first AR information in the AR video stream, and the first AR information is used to associate the character information with the real-world figure corresponding to the first story character for display.

[0036] In an optional design, the interactive module is used in at least one of the following ways:

[0037] Retrieve the character information of the first story character from the server;

[0038] The character information of the first story character is obtained through voice input;

[0039] The character information of the first story character is obtained by optical character recognition (OCR) scanning.

[0040] The character information of the first story character can be obtained by keyboard input.

[0041] In an optional design, the display module is used to display a first AR information control located around the real-world person in the AR video stream, the first AR information control displaying the first AR information.

[0042] In an optional design, the interactive module is used to obtain evidence-gathering information from the first story character;

[0043] The display module is used to display second AR information in the AR video stream, and the second AR information is used to display the evidence collection information of the first story character.

[0044] In an optional design, the display module is used to display a virtual scene related to the first story character; the interaction module is used to obtain first evidence information of the first story character in the virtual scene in response to a viewing operation of a specified location in the virtual scene.

[0045] Alternatively, the display module is used to display virtual props related to the first story character; the interaction module is used to obtain second evidence information associated with the first story character on the virtual props in response to an interaction operation with the virtual props.

[0046] Alternatively, the display module is used to display a non-player NPC story character related to the first story character; the interaction module is used to obtain third evidence information of the first story character in response to an interaction operation with the NPC story character.

[0047] In an optional design, the display module is used to display a second AR information control located around the real-world figure corresponding to the first story character in the AR video stream, the second AR information control displaying the second AR information; or, in the AR video stream, the second AR information control is displayed at the location where the evidence search information is acquired, the AR information control displaying the second AR information.

[0048] In an optional design, the display module is used to display a timeline control corresponding to the story scene. The timeline control displays at least one of the character information of the first story character and evidence collection information in chronological order. The interaction module is used to respond to the reasoning operation on the timeline control and collect evidence for the reasoning task corresponding to the story scene in the time dimension.

[0049] Alternatively, the display module is used to display at least one of the character information of the first story character and the evidence collection information; the interaction module is used to respond to the reasoning operation on the virtual map control and perform reasoning on the reasoning task corresponding to the story scene in the spatial dimension.

[0050] In an optional design, the display module is used to display task selection controls for at least two candidate story scenes;

[0051] In response to a selection operation on the task selection control, the selected story scene is determined from the at least two candidate story scenes.

[0052] In an optional design, the display module is used to display a character selection control for at least two candidate story characters in the story scene; in response to a selection operation on the character selection control, the selected story character is determined from the at least two candidate story characters;

[0053] The selected story character is bound to the facial data of the real-world person corresponding to the terminal.

[0054] In an optional design, the device further includes: an upload module for receiving an upload operation of the AR costume; and in response to the upload operation, uploading the locally created AR costume to a server.

[0055] In an optional design, the device further includes: a customization module for receiving customization operations of the AR outfit; and in response to the customization operations, uploading the customized AR outfit to a server.

[0056] According to one aspect of this application, a human-computer interaction device based on a story scenario is provided, the device being connected to a camera, the device comprising:

[0057] The receiving module is used to receive the real-scene video stream captured by the camera. The video frame of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person.

[0058] The processing module is used to process the real-world video stream into an augmented reality (AR) video stream. The video frame of the AR video stream includes an AR background area and an AR character area. The AR background area displays the scene frame of the story scene, and the AR character area displays the real-world character wearing AR costumes. The AR costumes correspond to the story characters in the story scene.

[0059] The interactive module is used to complete the reasoning task corresponding to the story scene based on the AR video stream;

[0060] The AR background area is obtained by processing the image content in the background area, and the AR character area is obtained by processing the image content in the foreground character area.

[0061] In an optional design, the processing module is used for:

[0062] When the real-world character in the real-world video stream is bound to the first story character in the story scene, the AR video stream is obtained; the AR character area in the AR video stream is the real-world character wearing the first AR outfit;

[0063] The first AR outfit includes the clothing corresponding to the first story character in the story scene.

[0064] In an optional design, the processing module is used for:

[0065] Image semantic recognition is performed on video frames in the real-scene video stream to obtain the background region and the foreground person region, wherein the foreground person region includes a face region and a non-face region;

[0066] Facial recognition is performed on the facial area to obtain the identity of the real-world person.

[0067] Based on the identity of the real-life character, determine the first story role that the real-life character is bound to in the story scene;

[0068] The background area is replaced or blended based on the scene materials of the story scene to obtain the AR background area; and the non-face area is replaced or blended based on the character materials of the first story character to obtain the AR character area;

[0069] The AR video stream is displayed based on the AR background area and the AR character area.

[0070] In an optional design, the processing module is used to process the second face area of ​​the AR character area to obtain a second face image of the real-world character without the AR terminal.

[0071] In an optional design, the processing module is used for:

[0072] The first facial image and the sample facial data of the real-world person are input into a generative network for image reconstruction to obtain a second facial image of the real-world person who is not wearing the AR terminal.

[0073] In the second face area of ​​the AR character area, the second face image is obtained through processing.

[0074] According to another aspect of this application, a terminal is provided, the terminal comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the story-scenario-based human-computer interaction method as described above.

[0075] According to another aspect of this application, a computer-readable storage medium is provided that stores a computer program, which is loaded and executed by a processor to implement the story-scenario-based human-computer interaction method as described above.

[0076] According to another aspect of this application, a computer program product is provided, the computer program product storing a computer program that is loaded and executed by a processor to implement the story-scenario-based human-computer interaction method as described above.

[0077] The beneficial effects of the technical solutions provided in this application include at least the following:

[0078] By replacing the background area in the real-world video stream with the AR background area corresponding to the reasoning task, and replacing the real-world characters in the video stream with real-world characters wearing AR costumes, it is possible to create story scenes for murder mystery games or escape rooms using AR scenes and AR costumes. This achieves a highly immersive visual effect without the need for specially customized costumes, props, or paper scripts. It delivers a superior visual experience compared to related technologies while reducing the consumption of specially customized physical resources. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 This illustration shows a schematic diagram of a real-world video stream captured by a user through a camera, provided in an exemplary embodiment of this application.

[0081] Figure 2 This illustration shows an AR video stream displayed on a user's AR terminal according to an exemplary embodiment of this application;

[0082] Figure 3 This application shows a structural block diagram of a computer system provided in an exemplary embodiment;

[0083] Figure 4 A flowchart illustrating a story-based human-computer interaction method provided in an exemplary embodiment of this application is shown.

[0084] Figure 5 A flowchart illustrating a method for displaying AR video streams based on real-scene video streams according to an exemplary embodiment of this application is shown.

[0085] Figure 6 This illustration shows a schematic diagram of an AR video stream in which real-world figures wearing AR glasses are not displayed in an exemplary embodiment of this application;

[0086] Figure 7 A flowchart of a human-computer interaction method based on a story scenario provided in another exemplary embodiment of this application;

[0087] Figure 8 A flowchart illustrating a method for obtaining evidence-gathering information of a first story character based on a story scene, provided in an exemplary embodiment of this application, is shown.

[0088] Figure 9 A flowchart illustrating a method for obtaining evidence-gathering information of a first story character based on a story scene, provided in another exemplary embodiment of this application, is shown.

[0089] Figure 10 A flowchart illustrating a method for obtaining evidence-gathering information of a first story character based on a story scene, provided in another exemplary embodiment of this application, is shown.

[0090] Figure 11 A flowchart illustrating a story-based human-computer interaction method provided in an exemplary embodiment of this application is shown.

[0091] Figure 12 A schematic diagram of dynamic video semantic segmentation provided in an exemplary embodiment of this application is shown;

[0092] Figure 13 A schematic diagram of an FCN network structure provided in an exemplary embodiment of this application is shown;

[0093] Figure 14 This illustration shows a diagram comparing the FCN semantic segmentation results provided by an exemplary embodiment of this application with the performance of real samples;

[0094] Figure 15 This illustration shows an AR screen displayed on a user's AR terminal during the evidence search phase, as provided in an exemplary embodiment of this application.

[0095] Figure 16 This invention illustrates a game scene flowchart of a story-based human-computer interaction method provided in an exemplary embodiment of this application.

[0096] Figure 17 A structural block diagram of a computer system provided in another exemplary embodiment of this application is shown;

[0097] Figure 18 A block diagram of a human-computer interaction device based on a story scenario provided in an exemplary embodiment of this application is shown;

[0098] Figure 19 A block diagram of a human-computer interaction device based on a story scenario provided in another exemplary embodiment of this application is shown;

[0099] Figure 20 This invention illustrates a structural block diagram of a terminal provided in an exemplary embodiment of this application;

[0100] Figure 21 A structural block diagram of a server provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Exemplary embodiments will be described in detail here, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0102] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0103] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another.

[0104] First, a brief introduction to the terms used in the embodiments of this application:

[0105] Deduction Missions: Also known as deduction games, these are tasks where one or more players solve puzzles based on clues within a story setting. Traditionally, story settings are primarily created through text on printed media. In recent years, however, the story settings in popular offline murder mystery games and escape room games are created by game venues set up by businesses.

[0106] Scripted Murder Mystery Games: Originating in the UK, this type of live-action role-playing game is based on the classic murder mystery game. The game revolves around a script, with a MC (Democrat) guiding the game's progress. Players complete their assigned story characters' deduction tasks through multiple rounds of evidence gathering, speaking, and reasoning, ultimately reconstructing the events (method of the crime). For example, in one script, a character might need to decipher the killer's method in a locked-room murder. The relationships between the various characters in the script are usually intricate, requiring players to immerse themselves in the story and carefully analyze the information gathered from other players' statements to ultimately identify the murderer. After the game, the MC reveals the truth and conducts a debriefing. Some scripts also trigger multiple endings based on player choices; these are called "mechanism-based" scripts.

[0107] The general game flow is as follows:

[0108] The host distributes scripts for different story characters to each player;

[0109] • The system will distribute items, which players can choose to trigger one of several endings;

[0110] Players introduce themselves based on the various story characters;

[0111] • Guided by the host, players gradually read the script, such as Act 1, Act 2, and so on;

[0112] • Evidence collection process and mechanism triggering process;

[0113] The murderer is chosen by a vote of all players;

[0114] • The host will review the plot – then the game ends;

[0115] • Typically, a game takes 4-5 hours to complete.

[0116] Escape Room: A type of real-life escape game. The earliest real-life escape room originated in 2006 when a group of system programmers in Silicon Valley designed a series of scenarios inspired by Agatha Christie's novels and brought them to life, providing them to all employees for adventure and puzzle-solving, naming it "Origin". The main ideas of the game are mostly derived from scenes in movies, TV series, books, and the internet. During the game, players usually take on the role of the protagonist from a first-person or third-person perspective and are confined to a nearly completely enclosed or threatening environment (i.e., the "escape room"). A single game has at least one escape room. Players need to find and use the props around them (such as paper props, mechanical props, electronic props, and live-action props), deduce and complete the assigned tasks (usually by solving specific puzzles) to ultimately escape the area.

[0117] Story Setting: Each reasoning task corresponds to a story, and the time, place, and environment of that story constitute the story setting of the reasoning task. For example, a Republican-era spy drama setting, a fantasy cultivation setting, a Western cowboy setting, and an ancient tomb exploration setting, etc.

[0118] Story Characters: Each reasoning task has at least one character. Different characters have different genders, appearances, personalities, backstories, plot progression roles, and reasoning tasks. These story characters can be fictional characters, fictional animals, anime characters, etc.

[0119] Gathering Evidence (referred to as evidence gathering): Each reasoning task has at least one piece of evidence. Different pieces of evidence have different plot-driving effects and are obtained by the player through corresponding actions on virtual props or other story characters.

[0120] Character Information: Different characters have different character information, such as name, age, gender, appearance, personality, background, social relationships, and schedule.

[0121] Public information refers to character information or evidence that all (or at least two) story characters in the reasoning task have viewing rights to access.

[0122] Private information refers to character information that only a specific story character has the right to view in a reasoning task. For example, among multiple pieces of information for the first story character, if information A can only be viewed by the second story character, then information A is the private information of the second story character.

[0123] This application provides an interactive solution for presenting a story scene to a user using AR for a reasoning task. This reasoning task can be a game task with a reasoning element, such as a murder mystery game or an escape room game. This application can use AR technology to present at least one of the following elements to the player: story scene, story characters, and character information.

[0124] Taking murder mystery game as an example, different players use different scripts and AR terminals. The AR terminal can be at least one of a smartphone, tablet, e-reader, laptop, desktop computer, and AR glasses. For example, users A, B, C, and D are sitting around a four-person table in a real-world setting, while user E stands beside the table. After starting the murder mystery game, users A, B, C, D, and E each possess an AR terminal. One of the five users (any user, the host user, or the administrator user) chooses a reasoning task from at least two candidate story scenarios. Figure 1 As shown, after performing image semantic recognition on the real-world scene, the AR terminal obtains a background area 101 and a foreground character area 102. The foreground character area 102 includes a face area 1021 and a non-face area 1022. After performing facial recognition on the face area 1021, the identity of the real-world character is obtained and bound to the story character selected by the player. For example, user A is bound to the first story character, user B to the second story character, user C to the third story character, user D to the fourth story character, and user E to the fifth story character.

[0125] Figure 1This diagram illustrates a real-world image 100 captured by user E's camera. The image includes a background area 101 and a foreground person area 102, corresponding to users A, B, C, and D. For example, the background area 101 shows the interior of a room with a low cabinet. The person areas include a face area 1021 and a non-face area 1022. User A faces sideways to user E's camera; the face area 1021 shows their true appearance and glasses, while the non-face area 1022 shows them with a bun hairstyle and wearing a sleeveless top. User B faces user E's camera; the face area 1021 shows their true appearance, while the non-face area 1022 shows them with bangs and shoulder-length hair, wearing a V-neck short-sleeved top. User C faces user E's camera; the face area 1021 shows their true appearance, while the non-face area 1022 shows them with shoulder-length hair and wearing a tank top. User D's camera is facing away from User E's; the non-face area 1021 shows that User D has slightly curly hair and is wearing a short-sleeved shirt.

[0126] Each user's AR terminal replaces the background area 101 in the real-world image 100 with the AR background area 201 based on the scene materials of the story scenario, and replaces the non-face area 1022 of the real-world character with the character materials of the story character, thereby replacing the foreground character area 102 with the AR character area 202, and thus replacing the real-world image 100 with the AR image 200. In at least one of the introduction phase and the evidence collection phase, the AR terminal will also display AR information controls 203 in the AR image.

[0127] After the reasoning task begins, players can obtain character information corresponding to other story characters based on AR screen 200. Character information can be obtained through at least one of the following methods: server acquisition, Optical Character Recognition (OCR) scanning, and keyboard input. For example, during public chat, private chat, or evidence-gathering phases, players can operate or interact with at least one of the story characters, AR props, and AR scenes. Their corresponding story characters will also perform the same actions, thereby obtaining evidence information about other story characters in the story scene, which is then displayed in AR information control 203. For example, based on AR screen 200, user D obtains basic character information of the bound fourth story character from the server: Character Four, female, 16 years old, daughter of Physician Yang; during public chat, users A, B, C, and E learn that Character Four had always been on good terms with the victim, but their relationship has recently deteriorated; after privately chatting with Character Four, user E learns that Character Four visited the crime scene yesterday; after searching Character Four's dressing case, user E discovers a damaged silver hairpin.

[0128] Figure 2An AR screen 200 displayed on user E's AR terminal is shown. This screen includes an AR background area 201, an AR character area 202, and an AR information control 203. The AR background area 201 displays the story scene of the reasoning task. The AR character area 202 displays a real-world character wearing AR costumes 204, which correspond to story characters in the story scene. The AR information control 203 displays evidence information (at least one of basic information, public information, and private information) of the story character. The AR information control 203 can be located around the story character or at the location where the evidence information is obtained. For example, the AR background area 201 shows a willow tree at the foot of a mountain outside the city. The AR character area 112 displays users A, B, C, and D wearing different ancient-style AR costumes, with user B playing an AR virtual guqin. Information about the fourth story character bound to user D is displayed on the AR information control 120 around him. The information obtained by User E during the private chat phase is private information and cannot be viewed by Users A, B, C, and D. The information obtained by User E during the evidence collection phase is made public to other players and can be viewed by Users A, B, C, and D.

[0129] After public chat, private chat, and evidence gathering, users A, B, C, D, and E voted to select the fourth story character associated with user D as the murderer. The voting result was correct, the reasoning task was completed, and users could choose a host to conduct a debriefing.

[0130] Figure 3 This illustration shows a structural block diagram of a computer system provided in an exemplary embodiment of this application. The computer system 300 includes a first terminal 310, a server 320, and a second terminal 330.

[0131] The first terminal 310 has a camera and has an application installed and running that supports AR interaction and reasoning tasks. The first terminal 310 is the first AR terminal used by a user.

[0132] The first terminal 310 is connected to the server 320 via a wireless network or a wired network.

[0133] Server 320 includes one of the following: a single server, multiple servers, a cloud computing platform, and a virtualization center. Illustratively, server 320 includes a processor 321 and a memory 322, with memory 322 further including a receiving module 3221, a display module 3222, and a control module 3223. Server 320 provides background services for applications supporting AR interaction and reasoning tasks. Optionally, server 320 undertakes the primary computing work, while the first terminal 310 and the second terminal 330 undertake secondary computing work; or, server 320 undertakes secondary computing work, while the first terminal 310 and the second terminal 330 undertake the primary computing work; or, server 320, first terminal 310, and second terminal 330 collaborate in a distributed computing architecture.

[0134] The second terminal 330 has a camera and has an application installed and running that supports AR interaction and reasoning tasks. The second terminal 330 is the AR terminal used by the second user.

[0135] Optionally, the first story character and the second story character are in the same story scene. Optionally, the first story character and the second story character can belong to the same team, the same organization, have a friend relationship, or have temporary communication permissions.

[0136] Optionally, the applications installed on the first terminal 310 and the second terminal 330 are the same, or the applications installed on the two terminals are the same type of application from different control system platforms. The first terminal 310 can refer to one of multiple terminals, and the second terminal 330 can refer to one of multiple terminals; this embodiment only uses the first terminal 310 and the second terminal 330 as examples. The device types of the first terminal 310 and the second terminal 330 may be the same or different, and these device types include at least one of: smartphones, tablets, e-book readers, laptops, desktop computers, and AR glasses. The following embodiments use terminals including smartphones and AR glasses as examples.

[0137] Those skilled in the art will understand that the number of terminals or story characters can be more or less. For example, there may be only one terminal or story character, or there may be dozens or hundreds, or even more. This application does not limit the number of terminals or story characters or the type of device.

[0138] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the real-scene video streams involved in this application are all obtained under full authorization, and the terminal and server only cache the real-scene video stream during program operation, and do not permanently store or reuse the relevant data of the real-scene video stream.

[0139] Figure 4 A flowchart illustrating a story-scenario-based human-computer interaction method provided in an exemplary embodiment of this application is shown. This embodiment uses this method... Figure 3 The terminal shown is used as an example. This terminal has a camera. The method includes:

[0140] Step 220: Acquire the real-scene video stream captured by the camera. The video stream includes the background area and the foreground person area.

[0141] The terminal has an application installed and running that supports AR interaction and reasoning tasks, which can be at least one of the reasoning games such as murder mystery or escape room games.

[0142] Taking murder mystery game scripts as an example, after the terminal receives the user's operation to launch the application, it displays at least two candidate story scenes. After the terminal receives the user's operation to select any story scene, it displays at least two candidate story characters. After the terminal receives the user's operation to select any story character, it also binds the user's facial data to the selected story character.

[0143] Taking escape room as an example, after the terminal receives the user's operation to launch the application, it displays at least two candidate story scenes. After the terminal receives the user's operation to select any story scene, it displays at least two candidate story characters. After the terminal receives the user's operation to select any story character, it also binds the user's facial data with the selected story character.

[0144] The terminal acquires a real-scene video stream captured by a camera. The real-scene video stream comprises multiple real-scene video frames, each constituting a real-scene video frame. These multiple real-scene video frames are arranged sequentially and displayed as a continuous video frame. In this embodiment, image semantic recognition is used to segment the real-scene video frames into background and foreground person areas, and these multiple real-scene video frames are arranged sequentially and displayed as a real-scene video stream. All or part of the real-scene video frames in this stream include both background and foreground person areas.

[0145] The background area refers to the scenery or set design that complements the real-world figures in the video frame captured by the terminal's camera. For example, the background area might be the walls and furniture of a room.

[0146] The foreground person area refers to the area of ​​the real-world person captured by the terminal's camera. The real-world person area includes the face area and non-face areas. The face area refers to the facial area of ​​the real-world person in the real-world image captured by the camera, while the non-face area refers to the area of ​​the real-world person in the real-world image captured by the camera other than the face area, such as the torso area and limbs area.

[0147] Image semantic recognition refers to the technology of using computers to process, analyze, and understand images in order to identify the two-dimensional regions containing different semantic objects within the same video frame or image. For example, distinguishing between background regions, face regions, and non-face regions within the same real-world video frame.

[0148] Step 240: Display the AR video stream based on the real-scene video stream;

[0149] An AR video stream comprises multiple AR video frames, each constituting an AR video frame. These AR video frames are arranged sequentially to form a continuous AR video frame. In some embodiments, there is a one-to-one correspondence between AR video frames and real-world video frames in the AR video stream. In some embodiments, there is a one-to-one correspondence between keyframes in the AR video stream and keyframes in the real-world video stream.

[0150] In this embodiment, the AR video frame includes an AR background area and an AR character area. The AR character area displays a real-world character wearing AR costumes, which correspond to story characters in the story scene. The AR background area is obtained by processing the image content in the background area, and the AR character area is obtained by processing the image content in the foreground character area.

[0151] AR background area: refers to the virtual background displayed when the application is running. This AR background area replaces the background area in the real-world video frame with scene materials based on the story scene, and merges the virtual content and the real-world content in real time to form an interactive three-dimensional environment between the virtual and the real.

[0152] AR Character Area: This refers to the area of ​​real-world characters wearing AR outfits displayed when the application is running. This AR character area replaces the non-facial areas of the real-world characters with the character materials of the story characters, and merges the virtual content and the facial areas of the real-world characters in real time to form a three-dimensional character area that interacts between the virtual and the real.

[0153] A reasoning task contains at least one story scene. Each story scene contains at least one story character. Each story character has at least one set of AR outfits. Optionally, a story character may have different AR outfits in different story scenes, a story character may have different AR outfits at different times within the same story scene, or a story character may have different AR outfits at different locations within the same story scene.

[0154] Step 260: Complete the reasoning task based on the AR video stream.

[0155] Based on AR information displayed in the AR video stream, complete at least one of the following tasks corresponding to the story scene: information acquisition, evidence collection, and puzzle reasoning. This AR information includes at least one of the following: AR information displayed around an AR character, AR information displayed around a virtual prop, AR information displayed in a virtual environment, and AR information displayed on a non-player character (NPC).

[0156] The AR information includes at least one of the following: text information, image information, video information, audio information, animation information, and special effects information related to completing the reasoning task.

[0157] The information acquisition task is used to obtain character information of each story character.

[0158] Evidence gathering tasks are tasks used to collect relevant evidence information for reasoning tasks.

[0159] Puzzle reasoning tasks are tasks that require solving puzzles based on the acquired character information and / or relevant evidence.

[0160] Taking a murder mystery game with a deduction task as an example, the deduction task includes at least one of the following: information gathering, evidence collection, and puzzle deduction. For illustration, the information gathering task includes at least one of the following stages: introduction, public chat, private chat, evidence gathering, and case closure. The evidence collection task includes the evidence gathering stage, and the puzzle deduction task includes the case closure stage.

[0161] In one embodiment, players obtain background information about the story during the script introduction phase, basic information about the story characters during the public chat phase, private extended information about the story characters during the private chat phase, and conduct evidence collection operations on virtual scenes, virtual props, or NPCs during the evidence collection phase to obtain evidence information. After analysis and voting, they obtain the reasoning result, which is displayed on the terminal or announced by the host to determine whether the result is correct or not, thus completing the reasoning task.

[0162] Taking an escape room puzzle as an example, the puzzle task includes at least one of the following: information gathering, evidence collection, and riddle-solving. Illustratively, the information gathering task includes an information searching phase, the evidence collection task includes an evidence gathering phase, and the riddle-solving task includes an escape phase. The information gathering phase, evidence gathering phase, and escape phase are also included. During the information gathering phase, players can learn about the story background, puzzle-solving task, or escape objective of the escape room through terminals, servers, or staff.

[0163] In one embodiment, players learn about the story background and escape objective of the escape room by reading virtual information controls displayed on the terminal. After searching for evidence in the virtual scene, virtual props, or NPCs during the evidence-gathering phase, they obtain the method to escape the escape room. After successfully escaping the escape room, they complete the reasoning task.

[0164] In summary, the method provided in this embodiment replaces the background area in the real-world video stream with the AR background area corresponding to the reasoning task, and replaces the real-world characters in the real-world video stream with real-world characters wearing AR costumes. This allows for the creation of story scenes for murder mystery games or escape rooms using AR scenes and AR costumes, providing a highly immersive visual effect without the need for specially customized costumes, props, and paper scripts. It achieves a superior visual experience compared to related technologies while reducing the consumption of specially customized physical resources.

[0165] In one possible embodiment, step 240 above includes steps 241, 243, 245, 247 and 249.

[0166] Figure 5 A flowchart illustrating a method for displaying AR video streams based on real-scene video streams according to an exemplary embodiment of this application is shown. This embodiment uses this method... Figure 3 The terminal and / or server shown are used to illustrate this method. The method includes:

[0167] Step 241: Perform image semantic recognition on the video frames in the real-scene video stream to obtain the background region and the foreground person region. The foreground person region includes the face region and the non-face region.

[0168] For example, users A, B, C, and D are sitting around a four-person table in a real-world scenario, while user E stands next to the table. The cameras of the AR terminals used by each user capture different real-world video streams, which are then uploaded to the server by each terminal. Each real-world video stream contains multiple real-world video frames. The server performs image semantic recognition on each real-world video frame to obtain the background area and foreground person area in each real-world video frame.

[0169] Taking the live video stream uploaded by user E's terminal as an example, the server can identify, after analysis, features such as... Figure 1 The real-scene video stream shown has a background area displaying the interior of the room and a low cabinet, and a foreground area displaying the faces, torsos, and limbs of users A, B, C, and D.

[0170] Step 243: Perform facial recognition on the facial area to obtain the identity of the real-world person;

[0171] Each face region of a real-world person identified by the server corresponds to a set of facial data. After the server performs identity recognition on the facial data, it can determine the identity of each real-world person.

[0172] For example, the facial regions of users A, B, C, D, and E each correspond to a set of facial data in the server. The AR terminal held by user E can capture the face of any one of these four users to obtain their corresponding real-world identity.

[0173] Step 245: Based on the real-life person's identity, determine the first story role that the real-life person is bound to in the reasoning task;

[0174] In an exemplary embodiment, before the reasoning task begins, users A, B, C, D, and E each use an AR terminal to select a story character, and the server binds their facial data to the story character.

[0175] In one exemplary embodiment, before the reasoning task begins, one of the five users (any user, host user, or administrator user) selects or assigns a story role, and the server binds each user's facial data to their corresponding story role.

[0176] The server stores the binding relationship between each user's facial data and the story characters. After the reasoning task starts, once the identity of the real-life person appearing in the real-life video frame is identified, the first story character bound to the real-life person in the reasoning task is determined based on this binding relationship.

[0177] Optionally, for other character identities, the second story character, third story character, etc., bound to the real-world character in the reasoning task can be determined based on this binding relationship.

[0178] Step 247: Replace the background area based on the scene material of the story scene to obtain the AR background area; replace the non-face area based on the character material of the first story character to obtain the AR character area, and obtain the AR video stream;

[0179] The server determines the scene assets for the story's scenarios, as well as the character assets corresponding to each story character.

[0180] In one embodiment, the server calculates the content in the background area of ​​the real-world video streams captured by users A, B, C, D, and E respectively, and renders the scene assets of the story scene onto the background area to obtain the AR background area. The server also calculates the content in the non-face area of ​​the foreground character area corresponding to each user, and renders the character assets of the bound story character onto the foreground character area to obtain a real-world character wearing AR attire, thus obtaining the AR character area. For example, the server renders the scene assets of the story scene onto the background areas of the real-world video streams captured by users A, B, C, D, and E respectively. Taking user A as an example, the server renders the character assets of the first story character bound to user A onto the non-face areas of user A captured by other users, thus obtaining the AR video stream.

[0181] In one embodiment, the server calculates the content in the background area of ​​the real-world video stream captured by user E, and renders the scene assets of the story scene into the background area to obtain the AR background area; the server 320 calculates the content in the non-face area of ​​the foreground character area corresponding to each user, and renders the character assets of the story character bound to that user into the foreground character area to obtain a real-world character wearing AR attire, thus obtaining the AR character area. For example, the server renders the scene assets of the story scene into the real-world video stream captured by user E, and renders the character assets of the story character bound to each user into the non-face area captured by user E, respectively, to obtain the AR video stream.

[0182] Optionally, the AR outfits for story characters can be created locally and uploaded to the server, or customized locally and uploaded to the server.

[0183] For example, AR costumes for story characters can be created locally, such as through modeling, and then uploaded to the server; or, AR costumes can be customized, such as by adjusting existing AR costumes, such as adjusting their size, shape, spacing, or color, and then uploaded to the server.

[0184] Step 249: Display the AR video stream based on the AR background area and the AR character area.

[0185] In one embodiment, the AR video stream displayed on each user's terminal is generated by their respective terminal's camera capturing real-world video streams, uploading them to the server for processing, and then the server transmits them back to their terminal. Taking user A as an example, the real-world video stream uploaded by user A's terminal is processed by the server to obtain an AR background area and an AR character area. The AR character area corresponds to users B, C, D, and E who are wearing AR costumes.

[0186] In one embodiment, the AR video stream displayed on each user's terminal is generated by the camera of any one terminal capturing a real-world video stream, uploading it to a server for processing, and then the server transmits it back to all terminals. For example, user E's terminal uploads a real-world video stream, which is processed to obtain an AR background area and an AR character area. The AR character area corresponds to users A, B, C, D, and E wearing AR costumes. Each terminal selectively displays the AR video stream from its corresponding perspective.

[0187] The above process can be executed multiple times by the computer system in a single reasoning task game. If the terminal has sufficient computing power, the entire process can also be executed by the terminal itself, without the need for a server.

[0188] In summary, the method provided in this embodiment identifies background and foreground person regions in a real-world video stream through image semantic segmentation. The foreground person regions include face regions and non-face regions.

[0189] In some embodiments, the AR terminal is AR glasses. Real-world figures displayed in a live video stream are wearing AR glasses. Since the AR glasses may not match the story scene—for example, if the story scene is a traditional Chinese fantasy setting—the appearance of AR glasses, as a modern consumer electronic device, does not conform to the traditional visual style. Therefore, this application provides a method for not displaying AR glasses worn by real-world figures in the AR video stream within a human-computer interaction method based on a story scene.

[0190] Figure 6 This illustration shows a method for not displaying real-world people wearing AR glasses in an AR video stream, as provided in an embodiment of this application. For example, the server inputs sample facial data of a real-world person collected and uploaded by the terminal, along with a first facial image 601 displayed in the AR video stream, into a generative network for image reconstruction, obtaining a second facial image 603 of the real-world person not wearing the AR terminal. This second facial image 603 is then displayed in the second facial region of the AR person area in the AR video stream.

[0191] In one example, a generative network is a neural network capable of image reconstruction. This generative network consists of a discriminator and a generator. During training, both the discriminator and generator need to be trained together; during application, only the generator is needed.

[0192] During training, the training set includes multiple sets of sample data from different users. Each set includes sample facial images (with the AR terminal worn) and sample facial data (without the AR terminal worn, such as a frontal image of the user) from the same user. The computer device inputs the sample facial images and sample facial data from the same user into the generator, which reconstructs the predicted facial image. The facial angles (which may be arbitrary) of the sample facial images and the predicted facial images are the same, but the predicted facial images do not show the user wearing the AR terminal. Optionally, the sample facial data is a frontal image of the user without the AR terminal worn, used to simulate the user's face captured during the binding phase. The facial angles of the sample facial images and the sample facial data can be different.

[0193] The discriminator is used to distinguish between predicted or sample facial images, identifying whether they are images reconstructed by the generator or the original images. An alternating training method based on the generative network is used: either the discriminator's network parameters are fixed while the generator's network parameters are updated; or the generator's network parameters are fixed while the discriminator's network parameters are updated. This continues until the error converges or the preset number of training iterations is reached, resulting in a trained discriminator and generator.

[0194] In the application phase, the computer device inputs the sample facial data of the real-world person collected and uploaded by the terminal, along with the first facial image displayed in the AR video stream, into the generator to reconstruct the image, thereby obtaining a second facial image of the real-world person without wearing the AR terminal.

[0195] Figure 7 A flowchart illustrating a method for human-computer interaction based on a story scenario provided in another exemplary embodiment of this application is shown. This embodiment uses this method... Figure 3 The terminal and / or server shown are executed to, as Figure 1 The example shown illustrates how users A, B, C, D, and E each possess a terminal performing the same inference task. Each terminal has a camera. The method includes:

[0196] Step 211: Display task selection controls for at least two candidate story scenarios;

[0197] A task selection control is used to select one story scene from at least two candidate story scenes. Task selection controls can be displayed as dropdown controls, card controls, or checkboxes, etc.

[0198] After receiving the user's command to start a reasoning task, the terminal displays a task selection control for at least two candidate reasoning tasks, with each reasoning task corresponding to a story scenario.

[0199] Taking a mobile phone as an example, users can perform touch operations (including but not limited to clicking, double-tapping, and swiping) on ​​the application to start a reasoning task. The application interface displays at least two candidate reasoning tasks, such as spy tasks, Western cowboy tasks, ancient fantasy tasks, and tomb exploration tasks. The cover of each reasoning task has a brief introduction to its story scene, which users can view by swiping the mobile phone interface.

[0200] Taking AR glasses as an example, users can use touch operations (including but not limited to clicking, double-clicking, dragging, and swiping) on ​​virtual selection controls that are floating in the air or laid out on a table in front of them to start reasoning tasks. The virtual selection controls display at least two candidate reasoning tasks, such as spy tasks, Western cowboy tasks, ancient fantasy tasks, and tomb exploration tasks. Each reasoning task's cover has a brief introduction to its story scene, which users can view by swiping or flipping through.

[0201] Step 212: In response to a selection operation on the task selection control, determine the selected story scene from at least two candidate story scenes;

[0202] The selection operation is used to select a story scene displayed in the task selection control. Selection can take the form of sliding a selection in a drop-down control, dragging a selection in a card control, or clicking a selection in a checkbox control.

[0203] Taking a mobile phone as an example, when user E clicks the selection control for the reasoning task "Ancient Style Story 1" on the interface of his mobile phone, the mobile phone interfaces of users A, B, C, D and E all display the story scene of "Ancient Style Story 1".

[0204] Taking AR glasses as an example, the virtual selection controls are laid out on the table in front of them. All five users can slide to view the various reasoning tasks. User E drags the "Ancient Style Story 1" selection control onto any user. The AR glasses of users A, B, C, D, and E all show that they have entered the story scene of "Ancient Style Story 1".

[0205] Step 213: Display a character selection control for at least two candidate story characters in the story scene;

[0206] A character selection control is used to select one story character from at least two candidate story characters. Character selection controls can be displayed as dropdown controls, card controls, or checkboxes, etc.

[0207] Taking a mobile phone as an example, the mobile phone interface displays at least 5 candidate story characters, such as: Character 1, Character 2, Character 4, Character 3 and Character 5.

[0208] Taking AR glasses as an example, the virtual selection control floats in the air, displaying virtual selection controls for at least 5 candidate story characters, such as: Character 1, Character 2, Character 4, Character 3, and Character 5.

[0209] Step 214: In response to a selection operation on the character selection control, determine the selected story character from at least two candidate story characters;

[0210] Taking a mobile phone as an example, each user selects a story character by clicking the story character selection control on their mobile phone screen. For example, user A clicks the character selection control, user B clicks the character selection control, user C clicks the character selection control, user D clicks the character selection control, and user E clicks the character selection control to complete the story character selection.

[0211] Taking AR glasses as an example, user A drags the virtual selection control of character one onto himself to select character one. User E drags the selection controls of character two, character three, character four and character five onto user B, user C, user D and himself respectively to complete the selection of story characters.

[0212] Step 215: Bind the facial data of the selected story character to the corresponding real-world figure on the terminal;

[0213] Taking a mobile phone as an example, each user uses the camera on their mobile phone to collect their own facial data.

[0214] In one embodiment, the mobile phone uploads the collected facial data to the server, and the server binds the facial data to a story character selected by the user. For example, user A's facial data is bound to character one, user B's facial data is bound to character two, user C's facial data is bound to character three, user D's facial data is bound to character four, and user E's facial data is bound to character five.

[0215] In one embodiment, the mobile phone calculates the collected facial data and binds the facial data to a story character selected by the user. For example, user A's facial data is bound to character one, user B's facial data is bound to character two, user C's facial data is bound to character three, user D's facial data is bound to character four, and user E's facial data is bound to character five.

[0216] Taking AR glasses as an example, and user E as an example, the AR glasses held by user E collect facial data from users A, B, C and D.

[0217] In one embodiment, user E's AR glasses upload the facial data of the four users to the server. The server then binds the facial data of user A, user B, user C, and user D to their selected roles one, two, three, and four, respectively. After user A's AR glasses collect user E's facial data, they upload it to the server. The server then binds user E's facial data to their selected role five.

[0218] In one embodiment, user E's AR glasses calculate the facial data of the four users and bind the facial data of user A, user B, user C, and user D to their selected roles one, two, three, and four, respectively. After user A's AR glasses collect user E's facial data, they calculate user E's facial data and bind the facial data to their selected role five.

[0219] In some embodiments, if five people other than the users suddenly enter the room during the reasoning task, the server will bind the facial data of that person, captured by the camera of any user's terminal, to an NPC in the story scene that does not advance the plot. For example, if a cleaning staff member suddenly enters the room, user E's camera captures the cleaning staff member's facial data and uploads it to the server via the AR terminal. The server then binds this facial data to the NPC sweeping maid in the story scene. The terminals of users A, B, C, D, and E will display the cleaning staff member as a maid wearing an ancient-style AR outfit sweeping the floor.

[0220] Step 220: Acquire the real-scene video stream captured by the camera. The real-scene video stream includes the background area and the foreground person area.

[0221] Taking user E as an example, the camera of their AR terminal collects data such as... Figure 1 The live video stream shown.

[0222] In one embodiment, after the real-scene video stream is uploaded to the server by the terminal, the server performs image semantic recognition to identify the background area and the foreground people area. The background area is a room with low cabinets and furnishings, and the foreground people area is users A, B, C and D showing their real appearances.

[0223] In one embodiment, after the camera captures a real-scene video stream, the terminal performs image semantic recognition on the real-scene video stream to identify the background area and the foreground people area. The background area is a room with low cabinets and furnishings, and the foreground people area is users A, B, C and D showing their real appearances.

[0224] Step 240: Display the AR video stream based on the real-scene video stream. The AR video stream includes an AR background area and an AR character area.

[0225] In one embodiment, each user's terminal replaces the scene and character assets of the story scene in the live video stream captured by its camera. Optionally, the scene or character assets may be obtained by the terminal from the server, or the scene or character assets may be read from the terminal.

[0226] In one embodiment, the server replaces the scene and character assets of the story scene with the real-world video stream uploaded by user E, and then sends the replaced AR video stream back to the terminals of the five users, with each terminal displaying the corresponding AR video stream from its own perspective.

[0227] Optionally, the scene material is a 3D scene that integrates virtual content and real-world content in real time, based on the time and space arrangement of the user's real environment; or, the scene material is a 3D scene that is inconsistent with the time and space of the user's real environment.

[0228] For example, users A, B, C, D, and E can all obtain scene materials (such as the foot of the mountain outside the city, Fujulou, and the bedrooms of characters 1, 2, 4, 3, and 5) and character materials (such as the clothing and accessories of characters 1, 2, 4, 3, and 5) from the server using their respective terminals. They then replace these scene and character materials in the captured real-world video stream. During the public chat phase, all five users are located in the scene at the foot of the mountain outside the city. The AR materials in this scene, such as mountains, water, trees, sky, and terrain, are inconsistent with the 3D structure of the users' actual rooms, creating a more expansive outdoor visual effect. During the evidence-gathering phase, the bedrooms of the five story characters are arranged according to the users' actual rooms; for example, virtual walls are rendered onto real walls, virtual beds are rendered onto real corners, and virtual tables and cabinets are rendered onto real tables and cabinets.

[0229] In the process of completing reasoning tasks corresponding to story scenes based on AR video streams, the reasoning tasks include at least one of the following: information acquisition tasks, evidence collection tasks, and puzzle reasoning tasks.

[0230] For information retrieval tasks:

[0231] Step 262: Obtain the character information of the first story character;

[0232] The terminal can obtain character information for the first story from at least one of the following methods: server access, voice input, OCR scanning, and keyboard input.

[0233] In one embodiment, the server stores character information for all story characters, and the terminal can automatically retrieve the corresponding character information from the server after the reasoning task enters a certain stage.

[0234] In one embodiment, after obtaining character information for the current story character or other story characters, the user can input that character information into the terminal via voice. For example, if the user has a paper script, they can read the text on the paper script aloud and input the character information of the story characters on the paper script into the terminal via voice.

[0235] In one embodiment, a user can use OCR to scan images, paper scripts, virtual paper props, etc., containing story character information to obtain the relevant information.

[0236] In one embodiment, a user can input character information of a story character into a terminal via a keyboard. This keyboard can be a keyboard displayed on the interface of a terminal with a camera, such as a smartphone, tablet, laptop, or e-book reader, or a virtual keyboard displayed by AR glasses.

[0237] For example, during the public chat phase, user E reads aloud the identity information and interpersonal relationship introduction of character five obtained from the server; user A and user C obtain the identity information and interpersonal relationship introduction of character five by voice recording user E's voice; user B obtains the identity information and interpersonal relationship introduction of character five by OCR scanning the information image on the desktop; and user D inputs the identity information and interpersonal relationship introduction of character five through the mobile phone keyboard or AR virtual keyboard.

[0238] For example, user E chats privately with user D and learns that user D's bound character four visited the crime scene at Fujiulou the day before the incident. This information is then entered into the AR terminal via the keyboard.

[0239] Step 264: Display the first AR information in the AR video stream. The first AR information is used to associate the character information with the real-world figures corresponding to the first story characters for display.

[0240] In one embodiment, user B's AR terminal displays a first AR information control located next to user A, and the role information about role one obtained by user B is displayed on this first AR information control. Public information about role one obtained by user B during the public chat phase can also be viewed by users C, D, and E on the first AR information control located next to user A displayed on their respective terminals; private information about role one obtained by user B during the private chat phase is not displayed on the first AR information control located next to user A displayed on the terminals held by users C, D, and E.

[0241] For example, a first AR information control about role five is displayed around user E, and information about role five obtained by other users during the reasoning task is displayed on the first AR information control.

[0242] For example, the information that user E obtained during the private chat that role four had been to the crime scene the day before the incident is displayed on the first AR information control located next to user D. Since this information is user E's private information, users A, B, C and D cannot see this information on the first AR information control next to user D.

[0243] For example, the acquired role information can be sorted according to the time the user acquired it or the timeline in the reasoning task, making it easier for the user to view, analyze, and reason.

[0244] For example, the information in the information control can be displayed in at least one of the following forms: text description, image description, voice description, or video playback.

[0245] In summary, the method provided in this embodiment offers a richer gaming experience by displaying reasoning tasks and story characters from at least two story scenarios and receiving user selection operations; it also provides a more immersive visual experience and sense of presence by binding story characters with the user's facial data and replacing the real-world video stream with an AR video stream; and by receiving the character information and evidence collection information of the first story character and displaying the character information and evidence collection information in the AR video stream, it reduces information recording costs, enhances the fun of information viewing, and facilitates users in binding story characters with information.

[0246] Regarding the evidence collection task:

[0247] Step 266: Obtain evidence information about the characters in the first story;

[0248] In one embodiment, a user can perform evidence-gathering operations on a virtual scene within the story scenario. These operations include, but are not limited to, touching, clicking, zooming in, zooming out, and stitching together, to obtain evidence information from the virtual scene.

[0249] In one embodiment, a user can perform evidence-gathering operations on virtual props in the story scene. These operations include, but are not limited to, opening, closing, breaking, piecing together, adjusting positions, and tapping, to obtain evidence information from the virtual props.

[0250] In one embodiment, a user can perform evidence-gathering operations on NPCs in the story scene. These operations include, but are not limited to, attacking, dodging, touching, hugging, and communicating with them to obtain evidence information from the NPCs.

[0251] Step 268: Display second AR information in the AR video stream, which is used to display evidence information of the first story character.

[0252] In one embodiment, the evidence obtained by user A in a virtual scene is displayed on a second AR information control within that virtual scene. If user A chooses to make it public, all story characters have viewing permission; if user A chooses to make it private, other story characters cannot view it.

[0253] In one embodiment, the evidence obtained by user B on a virtual prop is displayed on a second AR information control around the virtual prop. If user B chooses to make it public, all story characters have viewing permission; if user B chooses to make it private, other story characters cannot view it.

[0254] In one embodiment, the evidence information of a certain story character obtained by user C during the evidence-gathering phase is displayed on the second AR information control around the user. If user C chooses to make it public, all story characters have viewing permission; if user C chooses to make it private, other story characters cannot view it.

[0255] The above method for obtaining evidence-gathering information from the first story character based on the story scenario can be implemented in the following three ways:

[0256] The first scenario (evidence gathering based on a virtual scenario), such as Figure 8 As shown:

[0257] Step 266a: During the evidence-gathering phase, display virtual scenes related to the characters in the first story;

[0258] For example, after entering the evidence-gathering stage, users can freely choose the virtual scene to be searched. Optionally, there may be no NPCs in the virtual scene, or there may be NPCs in the virtual scene.

[0259] For example, if user E wants to search for evidence in character four's bedroom, they can select character four's bedroom in the AR virtual scene displayed on the AR terminal. This selection can be achieved by clicking, swiping, dragging, or fixing their gaze for more than 5 seconds.

[0260] Step 266b: In response to the evidence search operation in the virtual scene, obtain the first evidence search information of the first story character in the virtual scene;

[0261] For example, user C searches for evidence in the bedroom of role four and finds the furniture in disarray, as if it was messed up while searching for something. After user C's terminal is pointed at the messy furniture for 3 seconds, the terminal recognizes and obtains the evidence search information.

[0262] Step 268a: Display a second AR information control located around the real-world figure corresponding to the first story character in the AR video stream. The second AR information control displays the evidence search information of the first story character.

[0263] In one embodiment, user C discovers that the furniture in the bedroom of user D's character four is in disarray due to rummaging through things, and this evidence is displayed on the AR information control around user D.

[0264] The evidence collected can be selected to be public or private. If public is selected, other story characters in this story scene can also view it; if private is selected, other story characters in this story scene cannot view it.

[0265] The second scenario (using virtual props for evidence gathering), such as Figure 9 As shown:

[0266] Step 266c: During the evidence-gathering phase, display virtual props related to the characters in the first story;

[0267] For example, after entering the evidence collection stage, the user's AR terminal can display virtual props related to a certain story character. Optionally, the virtual props exist in a specific virtual scene, or the virtual props do not need to exist in a specific scene.

[0268] For example, if user E chooses to search for evidence on the guqin (a traditional Chinese stringed instrument) belonging to user B's second character, user E can choose to view the guqin in any scenario of the "Ancient Style Story 1" reasoning task.

[0269] For example, if user E chooses to search for evidence of the dressing room of character four bound to user D, the dressing room will only be displayed on user E's terminal after user E enters character four's bedroom.

[0270] Step 266d: In response to the search operation on the virtual prop, obtain the second search information associated with the first story character on the virtual prop;

[0271] For example, user E observes the AR virtual guqin of character two bound to user B and finds a bloodstain on the AR virtual guqin. After user E places his finger on the bloodstain for 3 seconds, the AR terminal recognizes and obtains the evidence information about character two.

[0272] For example, user E is conducting a search in the AR virtual bedroom of character four, which is bound to user D. After pointing the AR terminal at character four's dressing case for 5 seconds, the dressing case opens and a damaged silver hairpin is found inside. After pointing the AR terminal at the damaged silver hairpin for 3 seconds, the AR terminal recognizes and obtains the search information about character four.

[0273] Step 268b: In the AR video stream, a second AR information control is displayed at the location where the evidence is obtained. The second AR information control displays evidence information of the first story character.

[0274] In one embodiment, user E finds bloodstains on the guqin (a seven-stringed zither) belonging to character two bound to user B, and this evidence is displayed on an AR information control around the guqin.

[0275] In one embodiment, user B discovers a damaged silver hairpin in the dressing case of character four's bedroom, and this evidence is displayed on AR information controls around the dressing case.

[0276] The evidence collected can be selected to be public or private. If public is selected, other story characters in this story scene can also view it; if private is selected, other story characters in this story scene cannot view it.

[0277] The third scenario (based on NPC evidence collection), such as Figure 10 As shown:

[0278] Step 266e: During the evidence gathering phase, display NPCs related to the characters in the first story;

[0279] For example, after entering the evidence collection stage, the user's AR terminal can display an NPC related to a certain story character, which exists in a specific virtual scene, or the NPC does not need to exist in a specific virtual scene.

[0280] For example, if user B wants to search for evidence about the maid in character three's bedroom, they can select character three's bedroom in the AR virtual scene displayed on the AR terminal. This selection can be achieved by clicking, swiping, dragging, or fixing their gaze for more than 5 seconds. The bedroom will then display the presence of a maid.

[0281] Step 266f: In response to the interaction with the NPC story character, obtain the third evidence information associated with the first story character on the NPC story character;

[0282] For example, user B searches for evidence in the bedroom of character three, who is bound to user C. After pulling down the maid's sleeve, user B finds bruises. User B's terminal obtains the evidence information "character three often beats the maid" from the server.

[0283] Step 268c: In the AR video stream, a second AR information control is displayed at the location where the NPC is located. The second AR information control displays the evidence search information of the first story character.

[0284] In one embodiment, user B discovers injuries on the maid of user C (character three), and this evidence is displayed on the AR information control around user C.

[0285] The evidence collected can be selected to be public or private. If public is selected, other story characters in this story scene can also view it; if private is selected, other story characters in this story scene cannot view it.

[0286] For puzzle reasoning tasks:

[0287] In some embodiments, users can use a terminal to complete puzzle reasoning tasks based on AR video streams. The puzzle reasoning tasks based on AR video streams provided in this embodiment can be divided into the following two cases:

[0288] The first scenario, based on a timeline control:

[0289] Step a: Display the timeline control corresponding to the story scene. The timeline control displays at least one of the following in chronological order: character information of the first story character and evidence search information.

[0290] A timeline control is a control that can display character information or evidence collection information in chronological order. This chronological order can be the actual timeline or the timeline of story scenes.

[0291] For example, user A sorts the information about character two that has been acquired in the AR terminal according to the time sequence in the story scene on the timeline control displayed.

[0292] Step b: In response to the reasoning operation on the timeline control, perform reasoning on the reasoning task corresponding to the story scene in the time dimension.

[0293] Reasoning operations based on timeline controls include, but are not limited to, finding or reasoning out suspicious points on the timeline of the obtained character information or evidence information.

[0294] For example, user A finds evidence that character two's alibi was invalid in the information displayed in chronological order in the story scene.

[0295] The second scenario involves a virtual map control:

[0296] Step c: Display the virtual map control corresponding to the story scene. The virtual map control displays at least one of the following information according to the geographical location: character information of the first story character and evidence search information.

[0297] The virtual map control is a control that can display character information or evidence collection information according to geographical location. The virtual map can be a real geographical location or the geographical location of the story scene.

[0298] For example, user A displays the information about character three that has been acquired on the virtual map control displayed on the AR terminal according to the geographical location in the story scene.

[0299] Step d: In response to the reasoning operation on the virtual map control, reason about the reasoning task corresponding to the story scene in the spatial dimension.

[0300] Reasoning operations based on virtual map controls include, but are not limited to, finding or inferring suspicious points from the geographical locations of obtained character information or evidence information.

[0301] For example, user A finds evidence that character three visited the crime scene in the information of character three displayed according to the geographical location in the story scene.

[0302] Figure 11 A flowchart illustrating a story-scenario-based human-computer interaction method provided in an exemplary embodiment of this application is shown. This method can be executed by a terminal and a server. The method includes:

[0303] Step 1101: The terminal acquires the real-scene video stream captured by the camera;

[0304] The terminal is equipped with a camera. After the inference task is started, the camera on the terminal captures images of the surrounding environment, resulting in a live video stream. The live video stream consists of multiple live video frames (referred to as video frames) arranged in sequence.

[0305] Alternatively, the live video stream can be an encoded and compressed video stream.

[0306] Step 1102: The terminal sends the live video stream to the server;

[0307] The terminal sends the live video stream to the server via a wireless or wired network.

[0308] Step 1103: The server receives the live video stream reported by the terminal;

[0309] Step 1104: The server performs image semantic recognition on the real-scene video frames in the real-scene video stream to obtain the background area and foreground person area in the real-scene video frame; the foreground person area corresponds to the real-scene person.

[0310] The server stores an image semantic segmentation model. The server inputs the semantic segmentation result of the previous video frame and the current video frame into the image semantic segmentation model to obtain the semantic segmentation result of the current video frame. The semantic segmentation result includes the background region and the foreground person region.

[0311] Optionally, when processing the first video frame in the real-world video stream, the server inputs a reference segmentation result and the first video frame into an image semantic segmentation model to obtain the semantic segmentation result of the first video frame. This reference segmentation result can be a preset segmentation result, a coarse segmentation result obtained by semantically segmenting the first video frame using other models, or a blank segmentation result; this application does not limit the specifics of these options.

[0312] Optionally, when processing other video frames in the real-world video stream besides the first video frame, the server inputs the segmentation result of the (i-1)th video frame and the i-th video frame into the image semantic segmentation model to obtain the semantic segmentation result of the i-th video frame.

[0313] In one example, the image semantic segmentation model can output two semantic categories: background region and foreground person region; in another example, the image semantic segmentation model can output three semantic categories: background region, face region and non-face region in the foreground person region; in yet another example, the image semantic segmentation model can output multiple semantic categories: multiple sub-regions in the background region and foreground person region, which include at least two of the following: face region, torso region, limb region, palm region, finger region, and skeletal keypoints. This application does not limit the semantic segmentation capability of the image semantic segmentation model.

[0314] Step 1105: The server processes the image content in the background area to obtain the augmented reality (AR) background area, and processes the image content in the foreground character area to obtain the AR character area; the AR background area displays the scene of the story scene, and the AR character area displays real-world characters wearing AR costumes, with the AR costumes corresponding to the story characters in the story scene.

[0315] A reasoning task contains at least one story setting. Each story setting contains at least one story character. Each story setting has corresponding scene assets, and each story character has corresponding character assets.

[0316] Scene materials for the story setting include, but are not limited to, at least one of the following: natural environment materials, human-made building materials, outdoor decoration materials, indoor decoration materials, furniture materials, and environmental prop materials.

[0317] Character assets for story characters include, but are not limited to: jewelry assets, face makeup assets, top assets, trouser assets, dress assets, shoe assets, handheld prop assets, vehicle assets, or mount assets, etc. For example, the character assets for an ancient swordsman story character include jade hairpin assets, sect costume assets, and sword assets. Another example is a Western cowboy story character, whose character assets include cowboy hat assets, shirt assets, jeans assets, horse assets, and pistol assets, etc.

[0318] After identifying the background area and foreground character area in the real-world video frame, the server replaces or blends the background area based on the scene material of the story scene to obtain the AR background area; and replaces or blends the non-face area based on the character material of the first story character to obtain the AR character area.

[0319] In one example, the server directly replaces the background area with scene assets to obtain the AR background area without considering any physical information in the background area, such as replacing the background of an office in a real scene with the background of a mountain sect. In another example, the server considers the three-dimensional structural information in the background area, retains the original main structure in the background area, and uses surface maps from the scene assets to re-render the environment in the background area to obtain a personalized AR background area based on the original main structure, such as re-rendering a room in a real scene into the living room of a concubine.

[0320] In one example, the server replaces the clothing in the non-face area based on the character assets of the first story character to obtain the AR character area; or, the server adds virtual jewelry or virtual props to the non-face area based on the character assets of the first story character to obtain the AR character area.

[0321] Alternatively, since each real-life character can be associated with a different story role.

[0322] In some embodiments, the same story character has different AR outfits at different times in the same story scene; or, the same story character has different AR outfits at different locations in the same story scene; or, the same story character has different AR outfits in different story scenes; or, the same story character has different AR outfits at different times in different story scenes; or, the same story character has different AR outfits at different locations in different story scenes.

[0323] Step 1106: The server obtains an AR video stream based on the AR video frames after merging the AR background area and the AR character area;

[0324] Optionally, there is a one-to-one correspondence between AR video frames and real-world video frames. The AR background area and AR character area of ​​the same real-world video frame are merged to obtain the AR video frame corresponding to that real-world video frame.

[0325] The server stitches together the individual AR video frames in chronological order to obtain an AR video stream. Optionally, the server also encodes and compresses the AR video stream to reduce network bandwidth usage during data transmission.

[0326] Step 1107: The server sends an AR video stream to the terminal;

[0327] The server sends live video streams to the terminal via wireless or wired networks.

[0328] Step 1108: The terminal completes the reasoning task corresponding to the story scene based on the AR video stream.

[0329] In summary, the method provided in this embodiment, by having a cloud-based server undertake the computational task of image semantic segmentation, can significantly reduce the consumption of local computing resources on the terminal and provide a smoother AR experience.

[0330] In addition, in conjunction with references Figure 12 This embodiment implements dynamic video semantic segmentation based on traditional static semantic segmentation. In traditional static semantic segmentation, the image to be classified is input into an image semantic segmentation model for semantic segmentation to obtain the semantic segmentation result. In this embodiment, the semantic segmentation result of the previous video frame and the current video frame are input into the image semantic segmentation model for semantic segmentation to obtain the semantic segmentation result of the current video frame. Since the semantic segmentation result of the previous video frame is introduced as reference information in the image semantic segmentation task of the current video frame, the high temporal correlation between the previous and current video frames can be utilized to accelerate the calculation speed of the image semantic segmentation task, shorten the time spent on image semantic segmentation of the current video frame, and improve the accuracy of image semantic segmentation of the current video frame.

[0331] In some embodiments, the image semantic segmentation model is trained based on a basic sample library. The basic sample library includes: the semantic segmentation label of the previous sample video frame, the current sample video frame, and the semantic segmentation label of the current sample video frame. The previous sample video frame is the video frame preceding the current sample video frame in the sample video frame hierarchy. The semantic segmentation label of the previous sample video frame and the semantic segmentation label of the current sample video frame can be manually annotated matted sample masks, or matted sample masks obtained after semantic segmentation by a traditional static image semantic segmentation model.

[0332] In some embodiments, the image semantic segmentation model is trained using a base sample library and an enhanced sample library;

[0333] The basic sample library includes: the semantic segmentation label of the previous sample video frame, the current sample video frame, and the semantic segmentation label of the current sample video frame;

[0334] The enhanced sample library includes: the semantic segmentation label of the previous sample video frame, the current enhanced video frame, and the semantic segmentation label of the current enhanced video frame;

[0335] The current enhanced video frame is obtained by performing an affine transformation or thin-plate interpolation on the current sample video frame, and the semantic segmentation label of the current enhanced video frame is obtained by performing an affine transformation and / or thin-plate interpolation on the semantic segmentation label of the current sample video frame.

[0336] Given a limited number of samples in the basic sample library, to simulate scenarios such as human movement or rapid camera shake, the same pair of "current sample video frame + semantic segmentation label" is subjected to the same affine transformation or thin-plate interpolation to obtain a new pair of "current enhanced video frame + semantic segmentation label," thus forming an enhanced sample. By performing multiple different affine transformations or thin-plate interpolations, multiple enhanced samples can be obtained, forming an enhanced sample library.

[0337] Optionally, the server performs the same affine transformation or thin-plate interpolation on the background region in the same pair of current sample video frames and the semantic segmentation label of the current sample video frames to obtain the first enhanced sample; the server performs the same affine transformation or thin-plate interpolation on the foreground person region in the same pair of current sample video frames and the semantic segmentation label of the current sample video frames to obtain the second enhanced sample.

[0338] Based on Figure 11 In an optional embodiment, the above image semantic segmentation model can be implemented using a fully convolutional network (FCN).

[0339] Image semantic segmentation models need to determine the category of each pixel in an image. That is, image semantic segmentation is pixel-level. Previous convolutional networks (CNNs) used for semantic segmentation labeled each pixel with the category of the region surrounding it, but this method has significant drawbacks in both speed and accuracy. FCN is an improvement on CNN.

[0340] like Figure 13 As shown, the overall network structure of FCN is divided into two parts: a fully convolutional part and a deconvolutional part. The fully convolutional part borrows from some classic CNN networks (such as AlexNet, VGG, GoogLeNet, etc.), replacing the final fully connected layer with a convolutional layer. This convolutional layer is used to extract features and form a heatmap. The deconvolutional part upsamples the small-sized heatmap to obtain a semantic segmentation image of the original size. The input to the FCN network can be a color image of any size, and the output is the same size as the input, with n (number of object categories) + 1 (background) channels. The purpose of replacing fully connected layers with convolutional layers in the convolutional part of the CNN network is to allow the input image to be of any size exceeding a certain limit.

[0341] Because the heatmap becomes very small during convolution (e.g., its dimensions become the same as the original image), upsampling is needed to obtain dense pixel predictions of the original image size. An intuitive idea is to perform bilinear interpolation, which can be easily implemented using inverse convolution with a fixed kernel. Inverse convolution is also known as deconvolution or transposed convolution.

[0342] If we use the previously mentioned upsampling to segment the feature map of the last layer to the size of the original image, a lot of detail will be lost because the feature map of the last layer is too small. Therefore, we propose adding a skips structure to combine the predictions of the last layer (which has richer global information) with the predictions of shallower layers (which have more local details). This allows for local predictions while adhering to global predictions.

[0343] The predictions from the bottom layer (stride 32) (FCN-32s) are upsampled by a factor of 2 to obtain the original-size image, and then fused (added) with the predictions from pooling layer 4 (stride 16). This part of the network is called FCN-16s. Subsequently, this part of the predictions is upsampled by a factor of 2 again and fused with the predictions from pooling layer 3. This part of the network is called FCN-8s. (Refer to reference...) Figure 14 , Figure 14 The results show a comparison between FCN-32s, FCN-16s, FCN-8s, and real samples. It can be seen that the semantic segmentation result of FCN-8s is closest to the real sample, and the semantic segmentation result of FCN-8s is stronger than that of FCN-16s, while the semantic segmentation result of FCN-16s is stronger than that of FCN-32s.

[0344] FCN performs pixel-level image classification, thus solving the semantic-level image segmentation problem. Unlike classic CNNs, which use fully connected layers after convolutional layers to obtain fixed-length feature vectors for classification, FCN can accept input images of any size. It uses deconvolutional layers to upsample the feature map of the last convolutional layer, restoring it to the same size as the input image. This allows for a prediction for each pixel while preserving the spatial information of the original input image. Finally, it performs pixel-by-pixel classification on the upsampled feature map. The classification loss is then calculated pixel by pixel, meaning each pixel corresponds to a training sample.

[0345] Simply put, the difference between FCN and CNN is that FCN replaces the fully connected layer at the end of CNN with a convolutional layer, and the output is a pre-labeled image.

[0346] Figure 15The image shows an AR screen 300 displayed on the AR terminal of user E during the evidence-gathering phase. This screen includes an AR background area 301, an AR character area 302, and AR information controls. The AR background area 301 displays a virtual scene and virtual props 305 for the evidence-gathering phase. The AR character area 302 displays a real-world character wearing an AR costume 303, which corresponds to a story character in the story scene. The AR information controls 304 display character information or evidence-gathering information for the story character, and can be located around the real-world character to which the story character is bound or at the location where the evidence-gathering information is obtained.

[0347] For example, AR background area 301 shows a willow tree at the foot of a mountain outside the city. AR character area 302 shows user B wearing ancient-style AR clothing, and user B is playing an AR virtual guqin. The character information of the second story character obtained by user E is displayed on the AR information control 304 around user B. The private information of the second story character obtained by user E cannot be viewed by users A, B, C, and D, while the public information of the second story character obtained by user E can be viewed by users A, B, C, and D. The evidence information of the second story character obtained by user E after searching for evidence on the virtual guqin is displayed on the AR information control located at the evidence information acquisition position. This information can be selected to be public or private. If public is selected, other story characters in the story scene can also view it; if private is selected, other story characters in the story scene cannot view it.

[0348] Figure 16 This illustration shows a game scene flowchart of a story-based human-computer interaction method provided in an exemplary embodiment of this application. Figure 3 The terminal shown is used as an example. The terminal has a camera. The method includes: selecting a reasoning task 1601, selecting a story character 1602, reading the script 1603, an introduction stage 1604, a public chat stage 1605, a private chat stage 1606, an evidence gathering stage 1607, and a case closure stage 1608.

[0349] Select Reasoning Task 1601: The user selects any reasoning task from at least two candidate story scenarios displayed on the terminal.

[0350] For example, user A selects the current gunfight / espionage scenario from among the scenarios of fantasy cultivation, Western cowboy, and ancient tomb exploration.

[0351] Select Story Character 1602: After the user selects a story scene, they can choose any one of the two candidate story characters displayed on the terminal. The terminal camera captures the user's image and performs facial recognition to perform AR dressing up on the user and match the story character with the user.

[0352] For example, user A selects the first story character Agent A and completes the character binding to complete the AR outfit change.

[0353] Read Script 1603: Users read the background information of the selected story scene, and learn about the background, time, tasks, and basic information of the bound story characters, etc.

[0354] Introduction Phase 1604: Users introduce themselves to other story characters in the same story scene and obtain basic information about other story characters in the same story scene. This information is public information, and the information can be obtained through at least one of the following methods: obtaining from the server, voice input, OCR scanning, and keyboard input. This information is displayed in the AR information control, and users can scan the corresponding story character through their terminal device to view the AR information.

[0355] For example, user A introduces themselves to users in the same story scene and obtains basic information about users B and C from the server. This information is displayed in AR information controls located around users B and C. This information is public information, and users A, B, and C can scan the corresponding story characters with their terminal devices to view their basic information.

[0356] Public Chat Phase 1605: All users in the same story scene exchange information. Users can obtain extended information about story characters in the same story scene. This information is public information. The information can be obtained through at least one of the following methods: obtaining from the server, voice input, OCR scanning, and keyboard input. This information is displayed in the AR information control. Users can scan the corresponding story character through their terminal device to view the AR information.

[0357] For example, during a public chat, user A obtains the schedules of users B and C for the past three days using OCR scanning. This information is displayed in AR information controls located around users B and C. This information is public, and users A, B, and C can view this extended information by scanning the story characters bound to users B and C using their terminal devices.

[0358] Private Chat Stage 1606: Only two story characters in the same story scene exchange information. Users obtain extended information about the story characters they are chatting with. This information is private. The information can be obtained through at least one of the following methods: obtaining from the server, voice input, OCR scanning, and keyboard input. This information is displayed in the AR information control. Users can scan the corresponding story character with their terminal device to view the AR information.

[0359] For example, User A chats privately with User B and obtains extended information about User B's bound second story character by text input: User B's tool source is User C, and this information is displayed in an AR information control located next to User B. Only User A has viewing permission for this information, and User A can view this extended information by scanning User B's bound second story character with their terminal device.

[0360] Evidence Collection Phase 1607: Story characters perform evidence collection operations on virtual scenes or virtual props related to other story characters within the same story scene, obtaining evidence information about those other story characters. This information can be chosen to be public or private. If public, other story characters in the same story scene can also view it; if private, they cannot. Users can scan the corresponding story character or virtual prop using their terminal devices to view this evidence information.

[0361] For example, user A searches the desk of the third story character bound to user C to obtain a tool purchase and sale list. This information is displayed on an AR information control located at the desk. User A chooses to make the search information public. Users A, B, and C can all scan the third story character bound to user C or the desk with their terminal devices to view the search information.

[0362] Case Closure Phase 1608: Users A, B, and C vote, and the voting result is that User B is the target. The reasoning result is correct, the reasoning task is completed, and the case is closed.

[0363] In summary, the method provided in this embodiment utilizes a terminal to perform human-computer interaction and reasoning tasks, binds users to story characters using facial recognition and AR dress-up, and can obtain character information and evidence information from at least one of the following methods: server acquisition, voice input, OCR scanning, and keyboard input. The game operation is simple and convenient, providing a more immersive gaming experience.

[0364] Figure 17 This illustration shows a structural block diagram of a computer system provided in another exemplary embodiment of this application. This embodiment uses a method... Figure 3 The computer system shown is used as an example. The system includes: client 1701, background service 1702, architecture engine 1703, data storage 1704, and runtime environment 1705.

[0365] Client 1701: Refers to an Android or iOS application on a terminal that supports AR interaction and reasoning tasks. This terminal can be an electronic device with a camera, such as a smartphone, tablet, e-reader, laptop, desktop computer, or AR glasses. Client 1701 supports script selection, story character selection, and facial information input on the terminal. Client 1701 supports at least one AR function, including displaying AR scenes, AR clothing, and AR information. Client 1701 supports information recording, which can be done through at least one of OCR input, voice input, and keyboard input.

[0366] Background service 1702: Provided by server 320, it supports at least one of the following background services: data service, AR service, and intelligent input service, which are executed by client 1701. It intercepts and responds to requests from client 1701, filters and selects all requests from client 1701 or calls third-party interfaces, packages the information and then sends it back to client 1701.

[0367] Architecture Engine 1703: Performs operations such as application startup, request parameter processing, and response format rendering through the GIN framework, handles AR function operations through the AR engine, and handles machine learning computation operations through the AI ​​engine.

[0368] Data storage 1704: includes a MySQL database for storing general information and a MongoDB database for storing massive amounts of user logs and user image libraries. The two databases are stored independently and are deployed in a distributed cluster using Hadoop. The Distributed Relational Database Service (DRDS) is used as middleware to achieve elastic storage.

[0369] Operating Environment 1705: The background service 1702 uses a cloud computing platform to undertake the training tasks of the discriminator and generator based on the client dataset. It replaces the real-world video stream with the AR video stream through face recognition and image semantic recognition, and then transmits the images back to the Android or iOS client 1701 that supports AR interaction and reasoning tasks, providing users with a smoother and more immersive AR experience.

[0370] Those skilled in the art will understand that Figure 17 The computer architecture shown does not constitute a limitation on the computer system and may include more or fewer structures than shown, or combine certain structures, or adopt different structural arrangements.

[0371] Figure 18A block diagram of a story-based human-computer interaction device provided in an exemplary embodiment of this application is shown. The device includes: an acquisition module 1802, a display module 1804, a processing module 1806, and an interaction module 1808.

[0372] Acquisition module 1802: used to acquire the real-scene video stream captured by the camera. The video image of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person.

[0373] The device is equipped with and runs an application that supports AR interaction and reasoning tasks, which can be at least one of the reasoning games such as murder mystery or escape room games.

[0374] In some embodiments, after the user's facial data is bound to the selected story character, the acquisition module 1802 acquires the real-scene video stream captured by the camera. The real-scene video stream includes multiple real-scene video frames, each of which constitutes a real-scene video frame. The multiple real-scene video frames are arranged in chronological order and displayed as a continuous video frame.

[0375] The background area refers to the area of ​​scenery or set design that complements the real-world figures in the video frames captured by the acquisition module 1802. For example, the background area might be the walls and furniture of a room.

[0376] The foreground person area refers to the real-world person area captured by the acquisition module 1802. The real-world person area includes the face area and non-face areas. The face area refers to the face area of ​​the real-world person in the real-world image captured by the acquisition module 1802, while the non-face area refers to the area of ​​the real-world person in the real-world image captured by the acquisition module 1802 other than the face area, such as the torso area and limb areas.

[0377] Display module 1804: used to display the AR video stream based on the AR background area and the AR character area;

[0378] An AR video stream comprises multiple AR video frames, each constituting an AR video frame. These AR video frames are arranged sequentially to form a continuous AR video frame. In some embodiments, there is a one-to-one correspondence between AR video frames and real-world video frames in the AR video stream. In some embodiments, there is a one-to-one correspondence between keyframes in the AR video stream and keyframes in the real-world video stream.

[0379] In this embodiment, the AR video frame includes an AR background area and an AR character area. The AR character area displays a real-world character wearing AR costumes, which correspond to story characters in the story scene. The AR background area is obtained by processing the image content in the background area, and the AR character area is obtained by processing the image content in the foreground character area.

[0380] AR background area: refers to the virtual background displayed when the application is running. This AR background area replaces the background area in the real-world video frame with scene materials based on the story scene, and merges the virtual content and the real-world content in real time to form an interactive three-dimensional environment between the virtual and the real.

[0381] AR Character Area: This refers to the area of ​​real-world characters wearing AR outfits displayed when the application is running. This AR character area replaces the non-facial areas of the real-world characters with the character materials of the story characters, and merges the virtual content and the facial areas of the real-world characters in real time to form a three-dimensional character area that interacts between the virtual and the real.

[0382] A reasoning task contains at least one story scene. Each story scene contains at least one story character. Each story character has at least one set of AR outfits. Optionally, a story character may have different AR outfits in different story scenes, a story character may have different AR outfits at different times within the same story scene, or a story character may have different AR outfits at different locations within the same story scene.

[0383] In an optional design, the display module 1804 is used to display an AR video stream when the real-world character in the real-world video stream is bound to a first story character in the story scene; the AR character area in the AR video stream displays the real-world character wearing the first AR costume.

[0384] The first AR outfit includes the clothing worn by the first story character in the story scene.

[0385] In some embodiments, the display module 1804 displays an AR video stream, including an AR background area and an AR character area. The AR background area displays scene assets based on the story scenario, such as... Figure 2 The image shows the foot of the mountain outside the city and a virtual guqin (a seven-stringed zither) prop. The AR character area displays real-world characters wearing AR outfits, such as... Figure 2 The four players shown are dressed in ancient Chinese style AR outfits.

[0386] Display module 1804 is used to display first AR information in the AR video stream, the first AR information being used to display character information of a first story character. Display module 1804 is also used to display second AR information in the AR video stream, the second AR information being used to display evidence-gathering information of the first story character.

[0387] In an optional design, display module 1804 is used to display a first AR information control located around a real-world person in the AR video stream, the first AR information control displaying the first AR information. Display module 1804 is also used to display a second AR information control in the AR video stream, the second AR information control displaying second AR information.

[0388] The first or second AR information includes at least one of the following: text information, image information, video information, audio information, animation information, and special effects information related to completing the reasoning task.

[0389] In an optional design, the display module 1804 is used to display a second face image of the real-world person not wearing the AR device in the second face area of ​​the AR person area.

[0390] Processing module 1806:

[0391] Processing module 1806 is used to perform image semantic recognition on video frames in the real-scene video stream to obtain the background area and the foreground person area;

[0392] In this embodiment, the processing module 1806 segments the real-scene video frame into a background area and a foreground person area through image semantic recognition.

[0393] Image semantic recognition refers to the technology of using computers to process, analyze, and understand images in order to identify the two-dimensional regions containing different semantic objects within the same video frame or image. For example, distinguishing between background regions, face regions, and non-face regions within the same real-world video frame.

[0394] In some embodiments, after the processing module 1806 performs image semantic recognition on the video frames in the real-scene video stream, it obtains the following: Figure 1 The background area and foreground character areas are shown. The background area is the interior of the room and the low cabinet furnishings, while the foreground character areas are the facial and non-facial areas of users A, B, C, and D.

[0395] The processing module 1806 is also used to perform facial recognition on the facial region to obtain the identity of the real-world person; the processing module 1806 is also used to determine the first story role bound to the real-world person in the story scene based on the identity of the real-world person.

[0396] In some embodiments, after performing facial recognition on the facial regions of users A, B, C, and D, the processing module 1806 can obtain their identities. Based on the identities of users A, B, C, and D, the processing module 1806 can determine their associated story roles.

[0397] The processing module 1806 is also used to replace or merge the background area based on the scene material of the story scene to obtain the AR background area;

[0398] In some embodiments, the processing module 1806 replaces or blends the scene material of the story scene into the background area, resulting in, as shown below. Figure 2 The image shown is an AR background area displaying the foot of the mountain outside the city and a virtual guqin (a traditional Chinese stringed instrument).

[0399] The processing module 1806 is also used to replace or merge the non-face area based on the character material of the first story character to obtain the AR character area;

[0400] In some embodiments, the processing module 1806 replaces or merges the character assets of the story characters into the non-facial areas of users A, B, C, and D, resulting in, as shown below. Figure 2 The AR character area shown.

[0401] The processing module 1806 is also used to input the first facial image and the sample facial data of the real-world person into a generative network for image reconstruction, so as to obtain a second facial image of the real-world person who is not wearing the AR device.

[0402] In some embodiments, such as Figure 6 As shown, the processing module 1806 inputs the facial image shown in Figure 601 and the facial data of users A, B, C and D into the generative network for image reconstruction, and obtains the facial regions of users A, B, C and D who are not wearing AR glasses as shown in Figure 603.

[0403] Interactive module 1808: Used to complete the reasoning task corresponding to the story scene based on the AR video stream;

[0404] Reasoning tasks include, but are not limited to, information gathering tasks, evidence collection tasks, and puzzle reasoning tasks. Information gathering tasks are used to acquire character information about various story characters. Evidence collection tasks are used to collect relevant evidence information for reasoning tasks. Puzzle reasoning tasks are tasks that involve solving puzzles based on the acquired character information and / or relevant evidence information.

[0405] The interactive module 1808, based on the AR video stream displayed by the display module 1804, completes at least one of the following tasks corresponding to the story scene: information acquisition, evidence collection, and puzzle reasoning.

[0406] In an optional design, the interactive module 1808 is used to obtain character information of the first story character;

[0407] In an optional design, the interactive module 1808 is used in at least one of the following ways:

[0408] • Retrieve the character information of the first story character from the server;

[0409] • Obtain character information for the first story character via voice input;

[0410] • The character information of the first story character is obtained through optical character recognition (OCR) scanning.

[0411] • Obtain the character information of the first story character by keyboard input.

[0412] In an optional design, the display module 1804 is used to display a second AR information control located around the real-world figure corresponding to the first story character in the AR video stream, the second AR information control displaying the second AR information; or, in the AR video stream, the second AR information control is displayed at the location where the evidence search information is acquired, the AR information control displaying the second AR information.

[0413] In an optional design, the display module 1804 is used to display a timeline control corresponding to the story scene. The timeline control displays at least one of the character information of the first story character and evidence collection information in chronological order. The interaction module 1808 is used to respond to the reasoning operation on the timeline control and collect evidence for the reasoning task corresponding to the story scene in the time dimension.

[0414] Alternatively, the display module 1804 is used to display at least one of the character information of the first story character and the evidence search information; the interaction module 1808 is used to perform reasoning on the reasoning operation on the virtual map control in the spatial dimension for the reasoning task corresponding to the story scene.

[0415] In an optional design, the display module 1804 is used to display a virtual scene related to the first story character; the interaction module 1808 is used to obtain first evidence information of the first story character in the virtual scene in response to a viewing operation of a specified location in the virtual scene.

[0416] Alternatively, the display module 1804 is used to display virtual props related to the first story character; the interaction module 1808 is used to obtain second evidence information associated with the first story character on the virtual props in response to an interaction operation with the virtual props.

[0417] Alternatively, the display module 1804 is used to display a non-player NPC story character related to the first story character; the interaction module 1808 is used to obtain third evidence information of the first story character in response to an interaction operation with the NPC story character.

[0418] In an optional design, the interaction module 1808 is used to obtain evidence information from the first story character;

[0419] In an optional design, the display module 1804 is used to display task selection controls for at least two candidate story scenes; in response to a selection operation on the task selection controls, the selected story scene from the at least two candidate story scenes is displayed.

[0420] In an optional design, the display module 1804 is used to display a character selection control for at least two candidate story characters in the story scene; in response to a selection operation on the character selection control, the selected story character is determined from the at least two candidate story characters;

[0421] The processing module 1806 binds the selected story character with the facial data of the real-world person corresponding to the device.

[0422] In an optional design, the display module 1804 is used to perform at least one of the following tasks based on the AR information displayed in the AR video stream: information acquisition, evidence collection, and puzzle reasoning tasks corresponding to the story scene.

[0423] In an optional design, the device further includes: an upload module for receiving an upload operation of the AR costume; and in response to the upload operation, uploading the locally created AR costume to a server.

[0424] In an optional design, the device further includes: a customization module for receiving customization operations of the AR outfit; and in response to the customization operations, uploading the customized AR outfit to a server.

[0425] Figure 19 A block diagram of a story-based human-computer interaction device provided in an exemplary embodiment of this application is shown. The device includes a receiving module 1902, a processing module 1904, and an interaction module 1906.

[0426] The receiving module 1902 is used to receive the real-scene video stream captured by the camera. The video frame of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person.

[0427] In some embodiments, after a user's facial data is associated with a selected character in the story, a camera captures a live video stream. The live video stream comprises multiple live video frames, each constituting a live video frame. These multiple live video frames are arranged in chronological order and displayed as a continuous video frame.

[0428] The background area refers to the scenery or set design that complements the real-world figures in the video frame captured by the camera. For example, the background area might be the walls and furniture of a room.

[0429] The foreground person area refers to the area of ​​the real-world person captured by the camera. The real-world person area includes the face area and non-face areas. The face area refers to the face area of ​​the real-world person in the real-world image captured by the acquisition module 1802, while the non-face area refers to the area of ​​the real-world person in the real-world image captured by the acquisition module 1802 other than the face area, such as the torso area and limb areas.

[0430] The receiving module 1902 is also used to receive AR costumes created or custom AR costumes uploaded by the AR terminal.

[0431] Processing module 1904 is used to process the real-scene video stream into an augmented reality (AR) video stream. The video screen of the AR video stream includes an AR background area and an AR character area. The AR background area displays the scene screen of the story scene, and the AR character area displays the real-scene character wearing AR costumes. The AR costumes correspond to the story characters in the story scene.

[0432] The AR background area is obtained by processing the image content in the background area, and the AR character area is obtained by processing the image content in the foreground character area.

[0433] In an optional design, the processing module 1904 is used for:

[0434] When the real-world character in the real-world video stream is bound to the first story character in the story scene, the AR video stream is obtained; the AR character area in the AR video stream is the real-world character wearing the first AR outfit;

[0435] The first AR outfit includes the clothing corresponding to the first story character in the story scene.

[0436] In an optional design, the processing module 1904 is used for:

[0437] Image semantic recognition is performed on video frames in the real-scene video stream to obtain the background region and the foreground person region, wherein the foreground person region includes a face region and a non-face region;

[0438] Image semantic recognition refers to the technology of using computers to process, analyze, and understand images in order to identify the two-dimensional regions containing different semantic objects within the same video frame or image. For example, distinguishing between background regions, face regions, and non-face regions within the same real-world video frame.

[0439] Facial recognition is performed on the facial area to obtain the identity of the real-world person.

[0440] Based on the identity of the real-life character, determine the first story role that the real-life character is bound to in the story scene;

[0441] The background area is replaced or blended based on the scene materials of the story scene to obtain the AR background area; and the non-face area is replaced or blended based on the character materials of the first story character to obtain the AR character area;

[0442] The AR video stream is displayed based on the AR background area and the AR character area.

[0443] In some embodiments, after processing module 1904 performs image semantic recognition on video frames in the real-scene video stream, it obtains, as follows: Figure 1 The background area and foreground character areas are shown. The background area is the interior of the room and the low cabinet furnishings, while the foreground character areas are the facial and non-facial areas of users A, B, C, and D.

[0444] In some embodiments, after performing facial recognition on the facial regions of users A, B, C, and D, the processing module 1904 can obtain their identities. Based on the identities of users A, B, C, and D, the processing module 1904 can determine their associated story roles.

[0445] In some embodiments, the processing module 1904 replaces or blends the scene materials of the story scene into the background area, resulting in, as shown below. Figure 2 The image shown is an AR background area displaying the foot of the mountain outside the city and a virtual guqin (a traditional Chinese stringed instrument).

[0446] In an optional design, the processing module 1904 is used to process the second face area of ​​the AR character area to obtain a second face image of the real-world character without wearing an AR terminal.

[0447] In an optional design, the processing module 1904 is used for:

[0448] The first facial image and the sample facial data of the real-world person are input into a generative network for image reconstruction to obtain a second facial image of the real-world person who is not wearing an AR terminal.

[0449] In the second face area of ​​the AR character area, the second face image is obtained through processing.

[0450] In some embodiments, such as Figure 6 As shown, the processing module 1904 inputs the facial image shown in Figure 601 and the facial data of users A, B, C and D into the generative network for image reconstruction, and obtains the facial regions of users A, B, C and D who are not wearing AR glasses as shown in Figure 603.

[0451] Interactive module 1906 is used to complete the reasoning task corresponding to the story scene based on the AR video stream.

[0452] Reasoning tasks include, but are not limited to, information gathering tasks, evidence collection tasks, and puzzle reasoning tasks. Information gathering tasks are used to acquire character information about various story characters. Evidence collection tasks are used to collect relevant evidence information for reasoning tasks. Puzzle reasoning tasks are tasks that involve solving puzzles based on the acquired character information and / or relevant evidence information.

[0453] The interactive module 1906, based on the AR video stream processed by the processing module 1904, completes at least one of the following tasks corresponding to the story scene: information acquisition, evidence collection, and puzzle reasoning.

[0454] In an optional design, the interaction module 1906 is used to obtain character information of the first story character.

[0455] In an optional design, the interaction module 1906 is used to obtain evidence information from the first story character.

[0456] In an optional design, the interaction module 1906 is used to respond to a reasoning operation on the timeline control to search for evidence in the time dimension for the reasoning task corresponding to the story scene; or, the interaction module 1906 is used to respond to a reasoning operation on the virtual map control to search for evidence in the spatial dimension for the reasoning task corresponding to the story scene; or, the interaction module 1906 is used to respond to a viewing operation on a specified location in the virtual scene to obtain first evidence information of the first story character in the virtual scene; or, the interaction module 1906 is used to respond to an interaction operation on the virtual prop to obtain second evidence information associated with the first story character on the virtual prop; or, the interaction module 1906 is used to respond to an interaction operation on the NPC story character to obtain third evidence information of the first story character.

[0457] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules in human-computer interaction based on story scenarios. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For details of the specific implementation process, please refer to the method embodiments, which will not be repeated here.

[0458] Figure 20 A structural block diagram of a terminal 2000 provided in an exemplary embodiment of this application is shown. The terminal 2000 may be an electronic device with a camera, such as a smartphone, tablet computer, e-book reader, laptop computer, desktop computer, or AR glasses. The terminal 2000 may also be referred to as a user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0459] Typically, terminal 2000 includes a processor 2001 and a memory 2002.

[0460] Processor 2001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 2001 may also include a main processor and a coprocessor. The main processor, also known as the CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2001 may also include an AR processor for handling computational operations related to augmented reality. In some embodiments, processor 2001 may also include an AI (Artificial Intelligence) processor for handling computational operations related to machine learning.

[0461] The memory 2002 may include one or more computer-readable storage media, which may be non-transitory. The memory 2002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2002 are used to store at least one instruction, which is executed by the processor 2001 to implement the story-scenario-based human-computer interaction method provided in the method embodiments of this application.

[0462] In some embodiments, the terminal 2000 may also optionally include a peripheral device interface 2003 and at least one peripheral device. The processor 2001, memory 2002, and peripheral device interface 2003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 2003 via a bus, signal line, or circuit board. Specifically, the peripheral device may include at least one of the following: a radio frequency circuit 2004, a display screen 2005, a camera assembly 2006, an audio circuit 2007, and a power supply 2008.

[0463] Peripheral device interface 2003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 2001 and memory 2002. In some embodiments, processor 2001, memory 2002 and peripheral device interface 2003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 2001, memory 2002 and peripheral device interface 2003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0464] The radio frequency (RF) circuit 2004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 2004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 2004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 2004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 2004 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 2004 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0465] Display screen 2005 is used to display a UI (User Interface). This UI may include graphics, text, icons, video, and any combination thereof. When display screen 2005 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 2001 for processing. In this case, display screen 2005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 2005, which serves as the front panel of terminal 2000; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of terminal 2000 or in a folded design; in still other embodiments, display screen 2005 may be a flexible display screen, disposed on a curved or folded surface of terminal 2000. Furthermore, display screen 2005 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 2005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0466] The camera assembly 2006 is used to acquire images or videos. Optionally, the camera assembly 2006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 2006 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0467] The audio circuit 2007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 2001 for processing, or to the radio frequency circuit 2004 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 2000. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 2001 or the radio frequency circuit 2004 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 2007 may also include a headphone jack.

[0468] Power supply 2008 is used to power the various components in terminal 2000. Power supply 2008 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 2008 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0469] In some embodiments, the terminal 2000 further includes one or more sensors 2009. The one or more sensors 2009 include, but are not limited to: an accelerometer 2010, a gyroscope 2011, a pressure sensor 2012, an optical sensor 2013, and a proximity sensor 2014.

[0470] Accelerometer 2010 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 2000. For example, accelerometer 2010 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 2001 can control display screen 2005 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 2010. Accelerometer 2010 can also be used for games or for acquiring user motion data.

[0471] The gyroscope sensor 2011 can detect the orientation and rotation angle of the terminal 2000. The gyroscope sensor 2011, in conjunction with the accelerometer sensor 2011, can collect the user's 3D movements on the terminal 2000. Based on the data collected by the gyroscope sensor 2011, the processor 2001 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0472] The pressure sensor 2012 can be installed on the side bezel of the terminal 2000 and / or on the lower layer of the display screen 2005. When the pressure sensor 2012 is installed on the side bezel of the terminal 2000, it can detect the user's grip signal on the terminal 2000, and the processor 2001 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 2012. When the pressure sensor 2012 is installed on the lower layer of the display screen 2005, the processor 2001 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 2005. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0473] The optical sensor 2013 is used to collect ambient light intensity. In one embodiment, the processor 2001 can control the display brightness of the display screen 2005 based on the ambient light intensity collected by the optical sensor 2013. Specifically, when the ambient light intensity is high, the display brightness of the display screen 2005 is increased; when the ambient light intensity is low, the display brightness of the display screen 2005 is decreased. In another embodiment, the processor 2001 can also dynamically adjust the shooting parameters of the camera assembly 2006 based on the ambient light intensity collected by the optical sensor 2013.

[0474] The proximity sensor 2014, also known as a distance sensor, is typically installed on the front panel of the terminal 2000. The proximity sensor 2014 is used to detect the distance between the user and the front of the terminal 2000. In one embodiment, when the proximity sensor 2014 detects that the distance between the user and the front of the terminal 2000 is gradually decreasing, the processor 2001 controls the display screen 2005 to switch from a screen-on state to a screen-off state; when the proximity sensor 2014 detects that the distance between the user and the front of the terminal 2000 is gradually increasing, the processor 2001 controls the display screen 2005 to switch from a screen-off state to a screen-on state.

[0475] The memory further includes one or more programs stored in the memory, and the one or more programs include a signal display method based on a virtual environment provided in the embodiments of this application.

[0476] Those skilled in the art will understand that Figure 20 The structure shown does not constitute a limitation on the terminal 2000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0477] In an exemplary embodiment, a terminal is also provided, the terminal including a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is configured to be executed by the processor to implement the above-described human-computer interaction method based on a story scenario.

[0478] In an exemplary embodiment, a server 2100 is also provided, the server 2100 including a processor 2101 and a memory 2102. Figure 21 A structural block diagram of a server 2100 provided in an exemplary embodiment of this application is shown.

[0479] Typically, server 2100 includes a processor 2101 and memory 2102.

[0480] Processor 2101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2101 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 2101 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2101 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2101 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0481] The memory 2102 may include one or more computer-readable storage media, which may be non-transitory. The memory 2102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2102 is used to store at least one instruction, which is executed by the processor 2101 to implement the three-dimensional global illumination calculation method provided in the method embodiments of this application.

[0482] In some embodiments, server 2100 may optionally include an input interface 2103 and an output interface 2104. Processor 2101, memory 2102, and input interface 2103 and output interface 2104 can be connected via a bus or signal lines. Various peripheral devices can be connected to input interface 2103 and output interface 2104 via a bus, signal lines, or circuit board. Input interface 2103 and output interface 2104 can be used to connect at least one input / output (I / O) related peripheral device to processor 2101 and memory 2102. In some embodiments, processor 2101, memory 2102, and input interface 2103 and output interface 2104 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 2101, memory 2102, and input interface 2103 and output interface 2104 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0483] Those skilled in the art will understand that the structure shown above does not constitute a limitation on server 2100, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0484] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, a code set, or an instruction set. When executed by a terminal's processor, the at least one instruction, the at least one program, the code set, or the instruction set implements the perspective switching method in the virtual environment described above. Optionally, the computer-readable storage medium may be a ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, or optical data storage device, etc.

[0485] In an exemplary embodiment, a computer program product is also provided, which stores a computer program that is loaded and executed by a processor to implement the story-scenario-based human-computer interaction method as described above.

Claims

1. A human-computer interaction method based on a story scenario, characterized in that, The method is performed by a terminal equipped with a camera, and the method includes: The real-scene video stream captured by the camera is obtained. The video frame of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person. Image semantic recognition is performed on video frames in the real-scene video stream to obtain the background region and the foreground person region, wherein the foreground person region includes a face region and a non-face region; Facial recognition is performed on the facial area to obtain the identity of the real-world person. Based on the identity of the real-world character, determine the first story role that the real-world character is bound to in the story scene; The background area is replaced or blended based on the scene material of the story scene to obtain an AR background area, which displays the scene image of the story scene; and the non-face area is replaced or blended based on the character material of the first story character to obtain an AR character area, which displays the real-world character wearing a first AR outfit, the first AR outfit including the clothing corresponding to the first story character in the story scene. Based on the AR background area and the AR character area, display the AR video stream; The reasoning task corresponding to the story scene is completed based on the AR video stream.

2. The method according to claim 1, characterized in that, The first face area of ​​the foreground figure area displays the first face image of the real-world figure wearing an AR terminal; the method further includes: In the second face area of ​​the AR character area, a second face image of the real-world character without the AR terminal is displayed.

3. The method according to claim 2, characterized in that, The display of a second face image of the real-world person not wearing the AR terminal in the second face area of ​​the AR person area includes: The first facial image and the sample facial data of the real-world person are input into a generative network for image reconstruction to obtain a second facial image of the real-world person who is not wearing the AR terminal. The second face image is displayed in the second face area of ​​the AR character area.

4. The method according to claim 1, characterized in that, The reasoning task corresponding to the story scene based on the AR video stream includes: Based on the AR information displayed in the AR video stream, complete at least one of the following tasks corresponding to the story scene: information acquisition, evidence collection, and puzzle reasoning.

5. The method according to claim 4, characterized in that, The step of completing the information acquisition task corresponding to the story scene based on the AR information displayed in the AR video stream includes: Obtain the character information of the first story character; The AR video stream displays first AR information, which is used to associate the character information with the real-world figure corresponding to the first story character for display.

6. The method according to claim 5, characterized in that, Obtaining the character information of the first story character includes at least one of the following methods: Retrieve the character information of the first story character from the server; The character information of the first story character is obtained through voice input; The character information of the first story character is obtained by optical character recognition (OCR) scanning. The character information of the first story character can be obtained by keyboard input.

7. The method according to claim 5, characterized in that, The character information of the first story character includes at least one of public character information and private character information; The publicly available character information refers to the character information that at least two story characters participating in the story scene have viewing permissions for; The private character information refers to the character information that the second story character corresponding to the terminal has viewing permissions for.

8. The method according to claim 5, characterized in that, The step of displaying the first AR information in the AR video stream includes: A first AR information control located around the real-world figure is displayed in the AR video stream, and the AR information control displays the first AR information.

9. The method according to claim 4, characterized in that, The task of collecting evidence corresponding to the story scene based on the AR information displayed in the AR video stream includes: Obtain evidence information about the characters in the first story; The second AR information is displayed in the AR video stream. The second AR information is used to display the evidence collection information of the first story character.

10. The method according to claim 9, characterized in that, The step of obtaining evidence information about the first story character includes: Display a virtual scene related to the first story character; in response to a viewing operation at a specified location in the virtual scene, obtain first evidence information of the first story character in the virtual scene; or, Display virtual props associated with the first story character; in response to an interaction with the virtual props, obtain second evidence information associated with the first story character on the virtual props; or, Display non-player NPC story characters related to the first story character; in response to interaction with the NPC story characters, obtain third evidence information of the first story character.

11. The method according to claim 9, characterized in that, The step of displaying the second AR information in the AR video stream includes: A second AR information control is displayed in the AR video stream, located around the real-world figure corresponding to the first story character. The second AR information control displays the second AR information. or, In the AR video stream, a second AR information control is displayed at the location where the evidence search information is acquired, and the second AR information control displays the second AR information.

12. The method according to claim 4, characterized in that, The step of completing the puzzle reasoning task corresponding to the story scene based on the AR information displayed in the AR video stream includes: Displays a timeline control corresponding to the story scene, on which at least one of the character information of the first story character and evidence collection information is displayed in chronological order; in response to the reasoning operation on the timeline control, reasoning is performed on the reasoning task corresponding to the story scene in the time dimension; or, Display a virtual map control corresponding to the story scene. The virtual map control displays at least one of the character information of the first story character and evidence collection information according to geographical location. In response to the reasoning operation on the virtual map control, reasoning is performed on the reasoning task corresponding to the story scene in the spatial dimension.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: A task selection control that displays at least two candidate story scenarios; In response to a selection operation on the task selection control, the selected story scene is determined from the at least two candidate story scenes.

14. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Displays a character selection control for at least two candidate story characters in the story scene; In response to a selection operation on the character selection control, the selected story character is determined from the at least two candidate story characters; The selected story character is bound to the facial data of the real-world person corresponding to the terminal.

15. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Receive the upload operation of the AR outfit; In response to the upload operation, the locally created AR outfit is uploaded to the server.

16. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Receive the customization operation of the AR outfit; In response to the custom operation, the customized AR outfit is uploaded to the server.

17. A human-computer interaction method based on a story scenario, characterized in that, The method is executed by the server, and the method includes: The receiving terminal reports a real-scene video stream, the video frame of which includes a background area and a foreground person area, the foreground person area corresponding to the real-scene person; Image semantic recognition is performed on the real-scene video frames in the real-scene video stream to obtain the background region and the foreground person region, wherein the foreground person region includes a face region and a non-face region; Facial recognition is performed on the facial area to obtain the identity of the real-world person. Based on the identity of the real-world character, determine the first story role that the real-world character is bound to in the story scene; The background area is replaced or blended based on the scene material of the story scene to obtain an AR background area, which displays the scene image of the story scene; and the non-face area is replaced or blended based on the character material of the first story character to obtain an AR character area, which displays the real-world character wearing a first AR outfit, the first AR outfit including the clothing corresponding to the first story character in the story scene. An AR video stream is obtained based on the AR video frames after merging the AR background area and the AR character area; The AR video stream is sent to the terminal so that the terminal can complete the reasoning task corresponding to the story scene based on the AR video stream.

18. A human-computer interaction device based on a story scenario, characterized in that, The device has a camera, and the device includes: The acquisition module is used to acquire the real-scene video stream captured by the camera. The video frame of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person. The display module is configured to perform image semantic recognition on video frames in the real-scene video stream to obtain the background area and the foreground character area, wherein the foreground character area includes a face area and a non-face area; perform facial recognition on the face area to obtain the identity of the real-scene character; determine the first story character bound to the real-scene character in the story scene based on the identity of the real-scene character; replace or merge the background area based on the scene material of the story scene to obtain an AR background area, wherein the AR background area displays the scene image of the story scene; and replace or merge the non-face area based on the character material of the first story character to obtain an AR character area, wherein the AR character area displays the real-scene character wearing a first AR outfit, wherein the first AR outfit includes the clothing corresponding to the first story character in the story scene; and display the AR video stream based on the AR background area and the AR character area. The interactive module is used to complete the reasoning task corresponding to the story scene based on the AR video stream.

19. A human-computer interaction device based on a story scenario, characterized in that, The device is connected to a camera, and the device includes: The receiving module is used to receive the real-scene video stream captured by the camera. The video frame of the real-scene video stream includes a background area and a foreground person area, and the foreground person area corresponds to the real-scene person. The processing module is configured to perform image semantic recognition on video frames in the real-scene video stream to obtain the background region and the foreground character region, wherein the foreground character region includes a face region and a non-face region; perform facial recognition on the face region to obtain the identity of the real-scene character; determine the first story character bound to the real-scene character in the story scene based on the identity of the real-scene character; replace or merge the background region based on the scene material of the story scene to obtain an AR background region, wherein the AR background region displays the scene image of the story scene; and replace or merge the non-face region based on the character material of the first story character to obtain an AR character region, wherein the AR character region displays the real-scene character wearing a first AR outfit, wherein the first AR outfit includes the clothing corresponding to the first story character in the story scene; and display the AR video stream based on the AR background region and the AR character region. The interactive module is used to complete the reasoning task corresponding to the story scene based on the AR video stream.

20. A terminal, characterized in that, The terminal includes a camera, a processor, and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the character interaction method based on a story scene as described in any one of claims 1 to 16.

21. A server, characterized in that, The server includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the story-scene-based character interaction method as described in claim 17.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the story-scene-based character interaction method as described in any one of claims 1 to 16, or the story-scene-based character interaction method as described in claim 17.

23. A computer program product, characterized in that, The computer program product stores a computer program that is loaded and executed by a processor to implement the story-based character interaction method as described in any one of claims 1 to 16, or the story-based character interaction method as described in claim 17.

Images