Information processing method and device, computer equipment and storage medium
By automatically collecting environmental information in the vehicle and determining relevant scenarios, users can experience the target environment through simple selections, solving the problem of cumbersome operations in the existing technology and improving convenience and user experience.
Patent Information
- Application Number
- CN202411423606.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2025-01-21
AI Technical Summary
When a vehicle is in a specific environment, users need to perform multiple operations to feel the information of the environment, resulting in insufficient convenience.
By automatic collection of multimodal information of the target environment, determine multiple scenarios related to the target environment. Users only need to choose one scene to feel the environment of the scene and simplify the operation.
It improved the convenience of users to feel the environment, reduced the operation steps, enhanced the user experience, and avoided unnecessary information interference.
Smart Images
Figure CN119339308A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to information processing methods, devices, computer equipment and storage media. Background Art
[0002] When the vehicle is in the corresponding environment, the user in the vehicle often takes photos, records videos, and obtains information about the corresponding environment. When the user wants to experience the corresponding environment, the user performs operations on the information of the corresponding environment, such as opening pictures of the corresponding environment and opening videos of the corresponding environment to experience the corresponding environment. The user needs to perform more operations to experience the corresponding environment. How to improve the convenience of users experiencing the corresponding environment has become a problem that needs to be solved. Summary of the invention
[0003] In view of this, embodiments of the present disclosure provide an information processing method, apparatus, computer device, and storage medium.
[0004] In a first aspect, an embodiment of the present disclosure provides an information processing method, the method comprising:
[0005] Displaying scene identifiers of a plurality of first scenes related to a target environment, wherein the plurality of first scenes are determined from a scene set according to objects and association relationship information in the target environment, the association relationship information indicates a related object type of each scene in the scene set, the related object type of the scene is an object type related to the scene, each of the plurality of first scenes has multimodal information, the multimodal information of the first scene is determined from the multimodal information of the target environment, and the multimodal information of the target environment is collected by a target vehicle when the target vehicle is in the target environment;
[0006] determining a second scene selected by a first user in the target vehicle from a plurality of first scenes associated with the target environment;
[0007] An operation for simulating the second scene is performed using the multimodal information of the second scene.
[0008] In a possible implementation, the multimodal information of the target environment includes: a target environment image of the target environment, the multimodal information of the first scene includes: a scene image of the first scene, the scene image of the first scene includes: an object of each relevant object type of the first scene; and
[0009] Before displaying scene identifiers of a plurality of first scenes related to the target environment, the method further includes:
[0010] Objects irrelevant to the first scene are deleted from a target environment image including objects of each relevant object type of the first scene and objects irrelevant to the first scene to obtain a scene image of the first scene.
[0011] In a possible implementation, the multimodal information of the target environment includes: target environment audio of the target environment, the target environment audio includes: target environment audio frames, the multimodal information of the first scene includes: scene audio of the first scene, the scene audio of the first scene includes: scene audio frames of the first scene; and
[0012] Before displaying scene identifiers of a plurality of first scenes related to the target environment, the method further includes:
[0013] The audio signal irrelevant to the first scene is deleted from the target environment audio frame including the audio signal related to the first scene and the audio signal irrelevant to the first scene to obtain a scene audio frame of the first scene.
[0014] In a possible implementation, the multimodal information of the target environment includes: a target environment image of the target environment; and before displaying scene identifiers of a plurality of first scenes related to the target environment, further includes:
[0015] receiving an identifier of a second user-defined scene input by a second user in the target vehicle to determine the second user-defined scene;
[0016] determining an object selected by the second user from the displayed target environment image of the target environment, wherein the object selected by the second user is an object that the second user expects to appear in a scene customized by the second user;
[0017] The object type to which the object selected by the second user belongs is determined as a related object type of the scene customized by the second user, and the scene customized by the second user is added to the scene set.
[0018] In a possible implementation, before displaying scene identifiers of a plurality of first scenes related to the target environment, the method further includes:
[0019] Determine a plurality of candidate scenes corresponding to the environment type of the target environment from the scene set;
[0020] According to the objects and association relationship information in the target environment, a plurality of first scenes are determined from a plurality of candidate scenes.
[0021] In a possible implementation, determining multiple first scenes from multiple candidate scenes according to objects and association relationship information in the target environment includes:
[0022] Using a pre-trained language model for determining a first scene, multiple first scenes are determined from multiple candidate scenes corresponding to the environment type of the target environment according to prompt words including objects and association relationship information in the target environment.
[0023] In a second aspect, an embodiment of the present disclosure provides an information processing device, the information processing device comprising:
[0024] A display unit, used to display scene identifiers of a plurality of first scenes related to a target environment, wherein the plurality of first scenes are determined from a scene set according to objects and association relationship information in the target environment, the association relationship information indicates a related object type of each scene in the scene set, the related object type of the scene is an object type related to the scene, each of the plurality of first scenes has multimodal information, the multimodal information of the first scene is determined from the multimodal information of the target environment, and the multimodal information of the target environment is collected by a target vehicle when the target vehicle is in the target environment;
[0025] a determining unit, configured to determine a second scene selected by a first user in the target vehicle from a plurality of first scenes associated with the target environment;
[0026] An execution unit is used to utilize the multimodal information of the second scene to execute an operation for simulating the second scene.
[0027] In a possible implementation, the multimodal information of the target environment includes: a target environment image of the target environment, the multimodal information of the first scene includes: a scene image of the first scene, the scene image of the first scene includes: an object of each relevant object type of the first scene; the information processing device further includes:
[0028] A scene image acquisition unit is used to delete objects unrelated to the first scene from a target environment image including objects of each relevant object type of the first scene and objects unrelated to the first scene before displaying scene identifiers of multiple first scenes related to the target environment, so as to obtain a scene image of the first scene.
[0029] In a possible implementation, the multimodal information of the target environment includes: target environment audio of the target environment, the target environment audio includes: target environment audio frames, the multimodal information of the first scene includes: scene audio of the first scene, the scene audio of the first scene includes: scene audio frames of the first scene; the information processing device also includes:
[0030] A scene audio frame acquisition unit is used to delete the audio signal unrelated to the first scene from the target environment audio frame including the audio signal related to the first scene and the audio signal unrelated to the first scene before displaying the scene identifiers of multiple first scenes related to the target environment, so as to obtain the scene audio frame of the first scene.
[0031] In a possible implementation, the multimodal information of the target environment includes: a target environment image of the target environment; and the information processing device further includes:
[0032] A custom scene unit is used to receive an identifier of a second user-defined scene input by a second user in a target vehicle before displaying scene identifiers of multiple first scenes related to a target environment to determine a second user-defined scene; determine an object selected by the second user from a displayed target environment image of the target environment, wherein the object selected by the second user is an object that the second user expects to appear in the second user-defined scene; determine an object type to which the object selected by the second user belongs as a related object type of the second user-defined scene, and add the second user-defined scene to the scene set.
[0033] In a possible implementation, the information processing device further includes:
[0034] The first scene determination unit is used to determine multiple candidate scenes corresponding to the environment type of the target environment from the scene set before displaying the scene identifiers of multiple first scenes related to the target environment; and determine multiple first scenes from the multiple candidate scenes based on the objects and association relationship information in the target environment.
[0035] In one possible implementation, the first scene determination unit is further used to utilize a pre-trained language model for determining the first scene, and to determine multiple first scenes from multiple candidate scenes corresponding to the environment type of the target environment based on prompt words including objects and association relationship information in the target environment.
[0036] In a third aspect, an embodiment of the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0037] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding implementation manner thereof.
[0038] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method of the first aspect or any corresponding embodiment thereof.
[0039] The information processing method provided by the disclosed embodiment is that the target vehicle automatically collects multimodal information of the target environment and automatically determines multiple first scenes related to the target environment. The user in the target vehicle only needs to perform a simple operation, that is, select the second scene related to the target environment, and the user in the target vehicle can feel the second scene related to the target environment. Improve the convenience of the user to feel the environment. Considering that when the user expects to feel the corresponding environment, he specifically expects to feel the corresponding scene related to the corresponding environment, the user in the target vehicle can select the scene that the user expects to feel, that is, the second scene related to the target environment, and perform operations for simulating the second scene related to the target environment, so that the user in the target vehicle can feel the scene that the user in the target vehicle expects to feel, and avoid the situation where other information other than the information of the scene that the user in the target vehicle expects to feel when the user in the target vehicle feels the scene that he expects to feel may interfere with the user in the target vehicle, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0041] Figure 1 is a flowchart of an example of allowing a user in a vehicle to experience a scenario related to an environment;
[0042] Figure 2 is a flowchart of an information processing method provided by an embodiment of the present disclosure;
[0043] Figure 3 is a flowchart of another information processing method provided by an embodiment of the present disclosure;
[0044] Figure 4 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0046] refer to Figure 1 , which shows a flowchart of an example of allowing a user in a vehicle to experience a scene related to an environment.
[0047] In this example, the environment experienced by the user in the vehicle is a seaside environment.
[0048] Vehicle A collects multimodal information of the seaside environment when vehicle A is in the seaside environment.
[0049] As an example, vehicle A collects multimodal information of the seaside environment while traveling in the seaside environment.
[0050] As another example, vehicle A collects multimodal information of the seaside environment when it stops at a certain location in the seaside environment.
[0051] The multimodal information of the seaside environment includes: an image of the seaside environment. The image of the seaside environment is obtained by a camera of vehicle A.
[0052] As an example, the camera of vehicle A records a video to obtain a video of the beach environment. The video of the beach environment includes: a plurality of images of the beach environment.
[0053] As an example, the camera of vehicle A takes a photo of the seaside environment. As an example, the camera of vehicle A is a 360-degree panoramic camera, and the camera of vehicle A takes a photo of the seaside environment at each of multiple moments to obtain multiple images of the seaside environment. As an example, the camera of vehicle A takes a photo of the seaside environment at each of multiple preset shooting angles. Thus, multiple images of the seaside environment are obtained.
[0054] The multimodal information of the seaside environment also includes: temperature information of the seaside environment, wind volume of the seaside environment, and wind speed of the seaside environment.
[0055] When vehicle A is in the seaside environment, vehicle A collects temperature information through a temperature sensor on vehicle A, and the collected temperature information is used as the temperature information of the seaside environment.
[0056] When vehicle A is in the seaside environment, vehicle A collects odor information indicating odor through the odor sensor on vehicle A, and the collected odor information indicating odor is used as the odor information of the seaside environment.
[0057] When vehicle A is in the seaside environment, the wind volume is collected by a sensor on vehicle A for collecting wind volume, and the collected wind volume is used as the wind volume of the seaside environment.
[0058] The vehicle A's system determines the first scenes related to the seaside environment from the scene set, including "camping on the beach", "waves and beaches", "palm groves on the beach", "children playing with sand on the beach", "busy seaside road", and "sea and sky" based on the objects and association relationship information in the seaside environment.
[0059] When the user in vehicle A expects to experience scenes related to the seaside environment, the vehicle screen of vehicle A displays the logo of the scene of "camping on the beach", the logo of the scene of "waves and beach", the logo of the scene of "palm grove on the beach", the logo of the scene of "children playing with sand on the beach", the logo of the scene of "bustling seaside road", and the logo of the scene of "sea and sky".
[0060] It should be noted that the user in vehicle A may be a user who is in the vehicle when vehicle A is in the seaside environment. The user in vehicle A may also be a user who is not in vehicle A when vehicle A is in the seaside environment.
[0061] The user in vehicle A clicks on the icon of the scene "a busy seaside road" to select the scene "a busy seaside road".
[0062] The vehicle system of vehicle A can perform operations for simulating the scene of "a busy seaside road" based on the multimodal information of the scene of "a busy seaside road" determined from the multimodal information of the seaside environment, thereby allowing users in vehicle A to experience the scene of "a busy seaside road".
[0063] refer to Figure 2 , which shows a flow chart of the information processing method provided by the embodiment of the present disclosure. The method can be executed by a system of a target vehicle, such as a vehicle computer system. The target vehicle can be any vehicle to which the information processing method provided by the embodiment of the present disclosure can be applied.
[0064] In step S201, scene identifiers of a plurality of first scenes related to a target environment are displayed.
[0065] In the embodiment of the present disclosure, the target environment may be any environment to which the information processing method provided by the embodiment of the present disclosure may be applied. The scene related to the target environment may be referred to as the first scene related to the target environment.
[0066] When the user in the target vehicle desires to experience a scene related to the target environment, the scene identifiers of a plurality of first scenes related to the target environment may be displayed on the vehicle screen of the target vehicle.
[0067] In the embodiment of the present disclosure, a plurality of first scenes related to the target environment are determined from a scene set according to objects and association relationship information in the target environment.
[0068] As an example, for an object in the target environment, the object may be one of a scene, a person, an animal, and a plant.
[0069] It should be noted that, in the embodiments of the present disclosure, the object in the target environment may specifically refer to an object detected in the target environment.
[0070] The object in the target environment may be detected by performing target detection on a target environment image of the target environment through a trained target detection network such as Faster R-CNN.
[0071] It should be noted that, in the embodiment of the present disclosure, the scene in the scene set is a scene that may be related to the corresponding environment.
[0072] The scene set may include: a sub-scene set corresponding to each environment type in multiple environment types.
[0073] For an environment type, the scenes in the sub-scene set corresponding to the environment type are scenes that may be related to the environment of the environment type.
[0074] As an example, the sub-scene sets corresponding to the seaside environment type in the scene set include: "Seaside Camping", "Waves and Beach", "Palm Grove by the Sea", "Children Playing with Sand on the Beach", "Crowded Seaside Road", "Sea and Sky", etc. The sub-scene sets corresponding to the grassland environment type in the scene set include: "Grassland Horseback Riding", "Bonfire Party", "Grassland Dinner", etc.
[0075] In the disclosed embodiment, the multimodal information of the target environment includes: a target environment image of the target environment, a target environment audio of the target environment. The target environment audio of the target environment includes: a target environment audio frame of the target environment.
[0076] The multimodal information of the target environment also includes: other information related to the user's feelings besides the target environment image and the target environment audio of the target environment.
[0077] As an example, other information of the target environment related to the user's feelings includes: temperature information of the target environment, odor information of the target environment, wind volume of the target environment, etc.
[0078] In the disclosed embodiment, the multimodal information of the target environment is collected by the target vehicle when the target vehicle is in the target environment.
[0079] All target environment images of the target environment may belong to the target environment video of the target environment, and all target environment images of the target environment are arranged in the target environment video of the target environment in a sequence from front to back according to acquisition time.
[0080] All target environment images of the target environment may also be obtained by taking photos of the target environment respectively.
[0081] In the embodiment of the present disclosure, the association relationship information indicates the related object type of each scene in the scene set.
[0082] For a scene, the relevant object type of the scene is the object type related to the scene.
[0083] As an example, for the scene "camping by the sea", "sea type" and "tent type" are both related object types for the scene "camping by the sea". "Sea" and "tent" are both objects related to the scene "camping by the sea". For the scene "waves and beach", "wave type" and "beach type" are both related object types for the scene "waves and beach". "Waves" and "beach" are both objects related to the scene "waves and beach". For the scene "palm grove by the sea", "sea type" and "palm grove type" are both related object types for the scene. "Sea" and "palm grove" are both objects related to the scene "palm grove by the sea".
[0084] In the embodiment of the present disclosure, a plurality of first scenes related to the target environment are determined from a scene set based on objects and association relationship information in the target environment.
[0085] In one possible implementation, when multiple first scenes related to the target environment are determined from a scene set based on objects and association relationship information in the target environment, for a scene in the scene set, if it is determined that all objects in the target environment include objects of each relevant object type of the scene, then the scene can be determined as the first scene related to the target environment.
[0086] As an example, the target environment is a seaside environment. According to the fact that all objects in the seaside environment include objects of each relevant object type of the scene "camping at the seaside", i.e., objects such as the sea and tents, the scene "camping at the seaside" can be determined as the first scene related to the seaside environment. According to the fact that all objects in the seaside environment include objects of each relevant object type of the scene "waves and beaches", i.e., objects such as waves and beaches, the scene "waves and beaches" can be determined as the first scene related to the seaside environment. According to the fact that all objects in the seaside environment include objects of each relevant object type of the scene "palm groves at the seaside", i.e., objects such as the sea and palm groves, the scene "palm groves at the seaside" can be determined as the first scene related to the seaside environment. According to the fact that all objects in the seaside environment include objects of each relevant object type of the scene "sea and sky", i.e., objects such as the sea and sky, the scene "sea and sky" can be determined as the first scene related to the seaside environment.
[0087] In the embodiment of the present disclosure, the association relationship information may also indicate conditions corresponding to at least some of the scenes in the scene set.
[0088] As an example, for a scene in the scene set, the condition corresponding to the scene includes one or more of the following condition items: features of an object of a relevant object type of the scene, a minimum number of objects of the same relevant object type belonging to the scene, and a positional relationship between multiple objects of the relevant object type of the scene. As an example, features of an object of a character type related to the scene may include an action type of a character related to the scene and / or an expression type of a character related to the scene.
[0089] In another possible implementation, when multiple first scenes related to the target environment are determined from a scene set based on objects and association relationship information in the target environment, for a scene in the scene set, if it is determined that all objects in the target environment include objects of each relevant object type of the scene and based on the objects of the relevant object type of the scene in the target environment, it is determined that the conditions corresponding to the scene are met, then the scene can be determined as the first scene related to the target environment.
[0090] As an example, the target environment is a seaside environment. Based on the objects of the relevant object types of the scene "children playing with sand on the beach" in the seaside environment, i.e., children in the seaside environment, objects such as the beach in the seaside environment, and the action of children playing with sand in the seaside environment, the scene "children playing with sand on the beach" can be determined as the first scene related to the seaside environment. Based on the objects of the relevant object types of the scene "crowded seaside road" in the seaside environment, i.e., objects such as the road and people in the seaside environment, and the number of people on the road in the seaside environment is greater than the minimum number in the conditions corresponding to the scene "crowded seaside road", the scene "crowded seaside road" can be determined as the first scene related to the seaside environment.
[0091] In the embodiment of the present disclosure, each first scene related to the target environment has multimodal information.
[0092] Multimodal information of a first scene related to the target environment is determined from the multimodal information of the target environment.
[0093] The multimodal information of the target environment includes: one or more target environment images.
[0094] For a first scene related to the target environment, the multimodal information of the first scene includes: a scene image of the first scene and a scene audio of the first scene. The multimodal information of the first scene also includes: other information related to the user's feelings of the first scene in addition to the scene image of the first scene and the scene audio of the first scene. As an example, the other information related to the user's feelings of the first scene includes: temperature information of the first scene, odor information of the first scene, and air volume of the first scene.
[0095] It should be noted that, for a first scene related to the target environment, all scene images of the first scene may belong to the scene video of the first scene. If all scene images of the first scene belong to the scene video of the first scene, all scene images of the first scene are arranged in the scene video of the first scene in the order of collection time from early to late.
[0096] For a first scene related to the target environment, each scene image of the first scene includes: an object of each related object type of the first scene.
[0097] For a first scene related to the target environment, an object of a related object type of the first scene is an object related to the first scene.
[0098] In a possible implementation, for a first scene related to a target environment, in order to obtain a scene image of the first scene, a target environment image including objects of each relevant object type of the first scene can be determined. A target rectangular area in the target environment image including objects of each relevant object type of the first scene can be determined, wherein the target rectangular area is: the area with the smallest area among all areas surrounding the objects of each relevant object type of the first scene. A portion of the target environment image whose area is the target rectangular area can be cut out from the target environment image, and the portion of the target environment image whose area is the target rectangular area can be used as a scene image of the first scene.
[0099] In the embodiment of the present disclosure, an audio signal type that is unrelated to each scene in the scene set may be set in advance.
[0100] For a scene in the scene set, the type of the audio signal irrelevant to the scene is the type to which the audio signal irrelevant to the scene belongs.
[0101] In one possible implementation, for a first scene related to a target environment, in order to obtain a scene audio frame of the scene audio of the first scene, for each target environment audio frame in the target environment audio of the target environment, a neural network corresponding to an audio signal type unrelated to the first scene can be used to detect whether there is an audio signal unrelated to the first scene in the target environment audio frame. The neural network corresponding to the audio signal type unrelated to the first scene is trained to detect audio signals unrelated to the first scene. For a target environment audio frame, if an audio signal unrelated to the first scene is detected in the target environment audio frame, the volume of the audio signal related to the first scene in the target environment audio frame can be increased, and the volume of the detected audio signal unrelated to the first scene can be reduced. The audio signal related to the first scene in the target environment audio frame can be an audio signal in the target environment audio frame other than the audio signal unrelated to the first scene. By increasing the volume of the audio signal related to the first scene in the target environment audio frame and reducing the detected audio signal unrelated to the first scene, a volume-processed target environment audio frame corresponding to the target environment audio frame can be obtained, and the volume-processed target environment audio frame can be used as the scene audio frame of the scene audio of the first scene.
[0102] In an embodiment of the present disclosure, other information related to the user's feelings of the target environment may be used as other information related to the user's feelings of the first scene related to the target environment. As an example, other information related to the user's feelings of the target environment includes: temperature information of the target environment, odor information of the target environment, and wind volume of the target environment. Other information related to the user's feelings of the first scene related to the target environment includes: temperature information of the first scene, odor information of the first scene, and wind volume of the first scene. The temperature information of the target environment is used as the temperature information of the first scene. The odor information of the target environment is used as the odor information of the first scene. The wind volume of the target environment is used as the wind volume of the first scene.
[0103] In step S202 , a second scene selected by a first user in a target vehicle from a plurality of first scenes related to a target environment is determined.
[0104] The first user in the target vehicle can perform an operation of selecting a second scene from a plurality of first scenes related to the target environment.
[0105] As an example, the operation for the first user in the target vehicle to select the second scene is to click on the logo of the second scene.
[0106] In step S203, an operation for simulating the second scene is performed using the multimodal information of the second scene.
[0107] An operation for simulating the second scene is performed so that a user in the target vehicle can experience the second scene.
[0108] The multimodal information of the second scene may include: a scene image of the second scene, a scene audio of the second scene, and other information of the second scene related to the user's feelings. As an example, the other information of the second scene related to the user's feelings includes: temperature information of the second scene, odor information of the second scene, and air volume of the second scene.
[0109] It should be noted that all scene images of the second scene may belong to the scene video of the second scene.
[0110] The operation for simulating the second scene may include: displaying a scene image of the second scene; playing the scene audio of the second scene; controlling the scene simulating components on the target vehicle to form an object having other information items of the second scene related to the user's feelings. When all the scene images of the second scene belong to the scene video of the second scene, displaying the scene image of the second scene is playing the scene video of the second scene.
[0111] As an example, the target environment is a seaside environment. The first user in the target vehicle selects "waves and beach" as the second scene from the first scenes such as "camping at the beach", "waves and beach", "palm groves at the beach", "children playing with sand on the beach", "busy seaside roads", "sea and sky". The operations for simulating "waves and beach" include: playing the scene video of "waves and beach"; playing the scene audio of "waves and beach"; controlling the air conditioner of the target vehicle to output the wind volume in other information related to the user's feelings of the scene of "waves and beach"; controlling the temperature of the air conditioner of the target vehicle to the temperature indicated by the temperature information in other information related to the user's feelings of the scene of "waves and beach"; controlling the fragrance generator in the target vehicle to emit the smell indicated by the smell information in other information related to the user's feelings of the scene of "waves and beach". Thus, the user in the target vehicle can feel the scene of "waves and beach".
[0112] refer to Figure 3 , which shows a flow chart of another information processing method provided by an embodiment of the present disclosure.
[0113] In step S301, multiple first scenes related to the target environment are determined from a scene set based on objects and association relationship information in the target environment, and multimodal information of each first scene in the multiple first scenes is determined from the multimodal information of the target environment.
[0114] In the embodiment of the present disclosure, the multimodal information of the target environment includes: a target environment image of the target environment, a target environment audio of the target environment, and other information of the target environment related to the user's feelings.
[0115] All target environment images of the target environment may belong to the target environment video of the target environment obtained by video recording by the camera of the target vehicle. All target environment images of the target environment may also be obtained by taking pictures of the target environment by the camera of the target vehicle respectively.
[0116] In an embodiment of the present disclosure, for a first scene related to the target environment, the multimodal information of the first scene includes: a scene image of the first scene, a scene audio of the first scene, and other information of the first scene related to the user's feelings.
[0117] For a first scene related to the target environment, the scene image of the first scene includes: an object of each relevant object type of the first scene.
[0118] For a first scene related to the target environment, the object related to the first scene refers to an object of a related object type of the first scene.
[0119] In one possible implementation, for a first scene related to a target environment, in order to obtain a scene image of the first scene, objects unrelated to the first scene can be deleted from a target environment image that includes objects of each relevant object type of the first scene and objects unrelated to the first scene to obtain a scene image of the first scene.
[0120] The object irrelevant to the first scene is an object other than the object of the relevant object type of the first scene. In other words, the object irrelevant to the first scene is an object of an object type that is not the relevant object type of the first scene.
[0121] In the disclosed embodiment, objects unrelated to the first scene are deleted from a target environment image that includes objects of each relevant object type of the first scene and objects unrelated to the first scene to obtain a scene image of the first scene. This can prevent objects unrelated to the first scene from being included in the scene image of the first scene, thereby avoiding interference that objects unrelated to the first scene may cause to users in the target vehicle when the users in the target vehicle perceive the first scene, and further enhancing the experience of the users in the target vehicle in perceiving scenes related to the environment.
[0122] In the embodiment of the present disclosure, the target environment audio of the target environment includes: a target environment audio frame.
[0123] For a first scene related to the target environment, the scene audio of the first scene includes: a scene audio frame of the first scene.
[0124] In one possible implementation, for a first scene related to a target environment, in order to obtain a scene audio frame of the scene audio of the first scene, an audio signal unrelated to the first scene can be deleted from a target environment audio frame including an audio signal related to the first scene and an audio signal unrelated to the first scene to obtain the scene audio frame of the first scene.
[0125] The audio signal related to the first scene may be an audio signal in the target environment audio frame except for the audio signal unrelated to the first scene.
[0126] The audio signal that is not related to the first scene may be detected using a neural network corresponding to the type of the audio signal that is not related to the first scene.
[0127] In addition, for a first scene related to a target environment and a target environment audio frame, if the target environment audio frame does not contain an audio signal unrelated to the first scene, the target environment audio frame may be used as a scene audio frame of the first scene.
[0128] In the disclosed embodiment, audio signals irrelevant to the first scene are deleted from a target environment audio frame including an audio signal related to the first scene and an audio signal irrelevant to the first scene to obtain a scene audio frame of the first scene. This can ensure that the scene audio of the first scene does not include audio signals irrelevant to the first scene, thereby avoiding interference to users in the target vehicle caused by audio signals irrelevant to the first scene when users in the target vehicle experience the first scene, thereby further enhancing the experience of users in the target vehicle in experiencing scenes related to the environment.
[0129] In a possible implementation, multiple candidate scenes corresponding to the environment type of the target environment are determined from the scene set; and multiple first scenes related to the target environment are determined from the multiple candidate scenes based on objects and association relationship information in the target environment.
[0130] The candidate scenes corresponding to the environment type of the target environment are scenes that may be the first scenes related to the target environment.
[0131] As an example, the target environment is a seaside environment, and multiple candidate scenes corresponding to the seaside environment type include: "camping on the beach", "waves and beaches", "palm groves on the beach", "children playing with sand on the beach", "busy seaside road", and "sea and sky".
[0132] In the embodiment of the present disclosure, multiple first scenes are determined from a smaller number of scenes, that is, multiple candidate scenes. Compared with determining multiple first scenes from a scene set, the speed of determining multiple first scenes can be improved.
[0133] In a possible implementation, a pre-trained language model for determining the first scene is used to generate a prompt word including objects and association relationship information in the target environment. (P rompt ) , determining multiple first scenes from multiple candidate scenes.
[0134] The pre-trained language model for determining the first scene is trained.
[0135] As an example, prompt words including objects and association relationship information in the target environment include: "Refer to the objects and association relationship information in the target environment, determine multiple first scenes related to the target environment from the scene set, and feedback multiple first scenes. The following are some examples: Example 1: The reference information includes: object 1 in the seaside environment, object 2 in the seaside environment...the relevant object type 1 of the seaside environment, the relevant object type 2 of the seaside environment, and the feedback scenes related to the seaside environment include: "Camping at the beach", "Waves and beaches", "Palm groves at the beach", "Children playing with sand on the beach", "Crowded seaside road", "Sea and sky". Example 2:..."
[0136] In an embodiment of the present disclosure, a pre-trained language model for determining the first scene can be used to determine multiple first scenes from multiple candidate scenes. The pre-trained language model for determining the first scene has a high semantic understanding ability and can determine the first scene more accurately.
[0137] In a possible implementation, the scene can be customized by the second user in the target vehicle. It should be noted that the second user in the target vehicle may be the same as the first user in the target vehicle. The second user in the target vehicle may also be different from the first user in the target vehicle. The identification of the second user-defined scene input by the second user in the target vehicle can be received to determine the second user-defined scene; determine the object selected by the second user from the target environment image of the displayed target environment, wherein the object selected by the second user is the object that the second user expects to appear in the second user-defined scene; determine the object type to which the object selected by the second user belongs as the relevant object type of the second user-defined scene, and add the second user-defined scene to the scene set.
[0138] It should be noted that the displayed target environment image of the target environment may be a target environment video of the target environment. The displayed target environment image of the target environment may also be obtained by taking a photo of the target environment.
[0139] The operation of selecting an object on the displayed target environment image by the second user in the target vehicle may be clicking on the object in the displayed target environment image.
[0140] As an example, the target environment is a seaside environment, and the multiple scenes corresponding to the seaside environment type include: "camping on the beach", "waves and beaches", "palm groves on the beach", "children playing with sand on the beach", "crowded seaside roads", and "sea and sky". The second user-defined scene in the target vehicle is the scene of "playing a motorboat". The second user inputs the identifier of the second user-defined scene "playing a motorboat", so that the second user-defined scene can be determined to be the scene of "playing a motorboat". When playing the target environment video of the seaside environment, the second user clicks on the motorboat, the person on the motorboat, and the sea in the target environment image in the seaside environment target environment video. The motorboat, the person on the motorboat, and the sea are all objects selected from the target environment image of the displayed seaside environment target environment. The "motorboat type", "person type", and "sea type" are all determined as the relevant object types of the scene of "playing a motorboat". The scene of "playing a motorboat" is added to the scene set.
[0141] In an embodiment of the present disclosure, an identifier of a second user-defined scene input by a second user in a target vehicle is received to determine the second user-defined scene, and an object type to which an object selected by the second user belongs is determined as a relevant object type of the second user-defined scene, so that the second user can customize the scene and customize the relevant object type of the second user-defined scene. Thus, when it is determined that an environment is related to a user-defined scene, the user-defined scene can be used as the first scene related to the environment, and the multimodal information of the user-defined scene can be determined from the multimodal information of the environment. According to the multimodal information of the user-defined scene, the user-defined scene can be simulated. Thus, the user can experience the user-defined scene. The flexibility of the user to experience the scene related to the environment is improved.
[0142] In step S302, scene identifiers of a plurality of first scenes related to the target environment are displayed.
[0143] The process of step S302 refers to the process of step S201.
[0144] In step S303, a second scene selected by the first user from the plurality of first scenes is determined.
[0145] The process of step S303 refers to the process of step S202.
[0146] In step S304, an operation for simulating the second scene is performed using the multimodal information of the second scene.
[0147] The process of step S304 refers to the process of step S203.
[0148] The disclosed embodiments provide an information processing device. The information processing device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "unit" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0149] The information processing device comprises:
[0150] A display unit, used to display scene identifiers of a plurality of first scenes related to a target environment, wherein the plurality of first scenes are determined from a scene set according to objects and association relationship information in the target environment, the association relationship information indicates a related object type of each scene in the scene set, the related object type of the scene is an object type related to the scene, each of the plurality of first scenes has multimodal information, the multimodal information of the first scene is determined from the multimodal information of the target environment, and the multimodal information of the target environment is collected by a target vehicle when the target vehicle is in the target environment;
[0151] a determining unit, configured to determine a second scene selected by a first user in the target vehicle from a plurality of first scenes associated with the target environment;
[0152] An execution unit is used to utilize the multimodal information of the second scene to execute an operation for simulating the second scene.
[0153] In a possible implementation, the multimodal information of the target environment includes: a target environment image of the target environment, the multimodal information of the first scene includes: a scene image of the first scene, the scene image of the first scene includes: an object of each relevant object type of the first scene; the information processing device further includes:
[0154] A scene image acquisition unit is used to delete objects unrelated to the first scene from a target environment image including objects of each relevant object type of the first scene and objects unrelated to the first scene before displaying scene identifiers of multiple first scenes related to the target environment, so as to obtain a scene image of the first scene.
[0155] In a possible implementation, the multimodal information of the target environment includes: target environment audio of the target environment, the target environment audio includes: target environment audio frames, the multimodal information of the first scene includes: scene audio of the first scene, the scene audio of the first scene includes: scene audio frames of the first scene; the information processing device also includes:
[0156] A scene audio frame acquisition unit is used to delete the audio signal unrelated to the first scene from the target environment audio frame including the audio signal related to the first scene and the audio signal unrelated to the first scene before displaying the scene identifiers of multiple first scenes related to the target environment, so as to obtain the scene audio frame of the first scene.
[0157] In a possible implementation, the multimodal information of the target environment includes: a target environment image of the target environment; and the information processing device further includes:
[0158] A custom scene unit is used to receive an identifier of a second user-defined scene input by a second user in a target vehicle before displaying scene identifiers of multiple first scenes related to a target environment to determine a second user-defined scene; determine an object selected by the second user from a displayed target environment image of the target environment, wherein the object selected by the second user is an object that the second user expects to appear in the second user-defined scene; determine an object type to which the object selected by the second user belongs as a related object type of the second user-defined scene, and add the second user-defined scene to the scene set.
[0159] In a possible implementation, the information processing device further includes:
[0160] The first scene determination unit is used to determine multiple candidate scenes corresponding to the environment type of the target environment from the scene set before displaying the scene identifiers of multiple first scenes related to the target environment; and determine multiple first scenes from the multiple candidate scenes based on the objects and association relationship information in the target environment.
[0161] In one possible implementation, the first scene determination unit is further used to utilize a pre-trained language model for determining the first scene, and to determine multiple first scenes from multiple candidate scenes corresponding to the environment type of the target environment based on prompt words including objects and association relationship information in the target environment.
[0162] In this embodiment, the device is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and a memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0163] The further functional description of each of the above units is the same as that of the above corresponding embodiments and will not be repeated here.
[0164] refer to Figure 4, which shows a schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).
[0165] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0166] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0167] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0168] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0169] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means.
[0170] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0171] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0172] A portion of the embodiments of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can execute or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0173] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. An information processing method, characterized in that: The method comprises: Displaying scene identifiers of a plurality of first scenes related to a target environment, wherein the plurality of first scenes are determined from a scene set according to objects and association relationship information in the target environment, the association relationship information indicates a related object type of each scene in the scene set, the related object type of the scene is an object type related to the scene, each of the plurality of first scenes has multimodal information, the multimodal information of the first scene is determined from the multimodal information of the target environment, and the multimodal information of the target environment is collected by a target vehicle when the target vehicle is in the target environment; determining a second scene selected by a first user in the target vehicle from a plurality of first scenes associated with the target environment; An operation for simulating the second scene is performed using the multimodal information of the second scene.
2. The method according to claim 1, characterized in that The multimodal information of the target environment includes: a target environment image of the target environment, the multimodal information of the first scene includes: a scene image of the first scene, the scene image of the first scene includes: an object of each relevant object type of the first scene; and Before displaying scene identifiers of a plurality of first scenes related to the target environment, the method further includes: Objects irrelevant to the first scene are deleted from a target environment image including objects of each relevant object type of the first scene and objects irrelevant to the first scene to obtain a scene image of the first scene.
3. The method according to claim 1, characterized in that The multimodal information of the target environment includes: target environment audio of the target environment, the target environment audio includes: target environment audio frames, the multimodal information of the first scene includes: scene audio of the first scene, the scene audio of the first scene includes: scene audio frames of the first scene; and Before displaying scene identifiers of a plurality of first scenes related to the target environment, the method further includes: The audio signal irrelevant to the first scene is deleted from the target environment audio frame including the audio signal related to the first scene and the audio signal irrelevant to the first scene to obtain a scene audio frame of the first scene.
4. The method according to claim 1, characterized in that: The multimodal information of the target environment includes: a target environment image of the target environment; and Before displaying scene identifiers of a plurality of first scenes related to the target environment, the method further includes: Receiving an identifier of the second user-defined scene input by a second user in the target vehicle, so as to determine the second user-defined scene; determining an object selected by the second user from the displayed target environment image of the target environment, wherein the object selected by the second user is an object that the second user expects to appear in a scene customized by the second user; The object type to which the object selected by the second user belongs is determined as a related object type of the scene customized by the second user, and the scene customized by the second user is added to the scene set.
5. The method according to any one of claims 1 to 4, characterized in that Before displaying scene identifiers of a plurality of first scenes related to the target environment, the method further includes: Determine a plurality of candidate scenes corresponding to the environment type of the target environment from the scene set; The multiple first scenes are determined from the multiple candidate scenes according to the objects and association relationship information in the target environment.
6. The method according to claim 5, characterized in that Determining the plurality of first scenes from the plurality of candidate scenes according to the objects and the association relationship information in the target environment includes: The plurality of first scenes are determined from the plurality of candidate scenes according to prompt words including objects and association relationship information in the target environment using a pre-trained language model for determining the first scene.
7. An information processing device, characterized in that: The device comprises: A display unit, used to display scene identifiers of a plurality of first scenes related to a target environment, wherein the plurality of first scenes are determined from a scene set according to objects and association relationship information in the target environment, the association relationship information indicates a related object type of each scene in the scene set, the related object type of the scene is an object type related to the scene, each of the plurality of first scenes has multimodal information, the multimodal information of the first scene is determined from the multimodal information of the target environment, and the multimodal information of the target environment is collected by a target vehicle when the target vehicle is in the target environment; a determining unit, configured to determine a second scene selected by a first user in the target vehicle from a plurality of first scenes related to the target environment; An execution unit is used to utilize the multimodal information of the second scene to execute an operation for simulating the second scene.
8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 6 by executing the computer instructions.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.
10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.