Equipment information processing method, electronic equipment and storage medium

Abstract

The invention provides an equipment information processing method, electronic equipment and a storage medium, and the method comprises the steps: responding to a received triggering instruction, and determining a virtual scene associated with the triggering instruction; wherein the virtual scene comprises a virtual space and at least one virtual device in the virtual space, and the at least one virtual device corresponds to the at least one real device; outputting a virtual scene; and in response to a received processing instruction for target virtual equipment in the at least one virtual equipment from the object, processing equipment information of target real equipment corresponding to the target virtual equipment in the at least one real equipment based on the processing instruction to obtain a processing result.

Description

Device information processing method, electronic device, and storage medium Technical Field

[0001] The present disclosure relates to technical fields such as the metaverse, the Internet of Things, and virtual simulation, and relates to a device information processing method, apparatus, electronic device, storage medium, and computer program product. Background Art

[0002] Buildings are often equipped with multiple home appliances, such as air conditioners and lighting. Users need to use remote controls or control panels on these appliances to check or control their operating status. However, sometimes users are away from home, making it impossible to check or control the operating status of their appliances.

[0003] Summary of the Invention

[0004] The present disclosure provides a device information processing method, apparatus, electronic device, storage medium, and computer program product.

[0005] According to one aspect of the present disclosure, a device information processing method is provided, comprising: in response to receiving a trigger instruction, determining a virtual scene associated with the trigger instruction; wherein the virtual scene includes a virtual space and at least one virtual device in the virtual space, and the at least one virtual device corresponds to at least one real device; outputting the virtual scene; and in response to receiving a processing instruction from an object for a target virtual device in at least one virtual device, processing device information of the target real device corresponding to the target virtual device in the at least one real device based on the processing instruction to obtain a processing result.

[0006] In this embodiment, the processing instruction includes a query instruction; processing the device information of the target real device corresponding to the target virtual device in at least one real device includes: based on the identification information in the query instruction, querying the device information for the target virtual device from the database; the device information of the target virtual device represents the status of the real device related to the target virtual device.

[0007] In this embodiment, the above method also includes: obtaining identification information and device information of the real device based on at least one of a third-party platform, a third-party application, a terminal device and a real device; and storing the identification information, device information and mapping information in a database; wherein the mapping information represents the correspondence between the identification information and device information of the real device.

[0008] In this embodiment, the processing instruction includes a control instruction; and processing the device information of the target real device corresponding to the target virtual device in the at least one real device includes: controlling the operating state of the target real device based on the control instruction.

[0009] In this embodiment, determining the virtual scene associated with the trigger instruction includes: determining the virtual scene associated with the object identifier based on the object identifier in the trigger instruction.

[0010] In this embodiment, the above method also includes: acquiring multiple real space images obtained by image acquisition for the real space; wherein the real space includes at least one real device, and the multiple real space images include a device image for the at least one real device; determining the category of the at least one real device based on the device image of the at least one real device; and determining a virtual device corresponding to the at least one real device from a material library based on the category of the at least one real device; and adding the virtual device determined from the material library to the virtual space.

[0011] In this embodiment, the above method also includes: obtaining multiple device images obtained by image capture of real devices; generating a virtual device corresponding to the real device based on the multiple device images; and adding the generated virtual device to the material library, which stores a three-dimensional model, and the three-dimensional model represents the virtual space and virtual device.

[0012] In this embodiment, the real device includes at least one device sub-area distributed along a predetermined direction, and the multiple device images are obtained by performing ambient photography on each device sub-area.

[0013] In this embodiment, the method further includes: in response to receiving a space modification instruction for the initial virtual space from the object, modifying the initial virtual space based on the space modification instruction; and determining the modified initial virtual space as the virtual space.

[0014] In this embodiment, the method further includes: in response to receiving a device modification instruction for at least one initial virtual device from the object, modifying the at least one initial virtual device based on the device modification instruction; and determining the modified initial virtual device as the virtual device.

[0015] In this embodiment, the method further includes: in response to receiving an adding instruction for at least one candidate virtual device in the material library from the object, adding the at least one candidate virtual device as a virtual device to the virtual space.

[0016] In this embodiment, the above method also includes: in response to receiving a device processing instruction from the object, processing the virtual device in the virtual space based on the processing mode in the device processing instruction; wherein the processing mode includes at least one of the following: deleting the virtual device, modifying the virtual image of the virtual device, and modifying the attribute information of the virtual device.

[0017] In this embodiment, the virtual space corresponds to the real space, and at least a portion of the at least one real device is located in the real space.

[0018] In this embodiment, the real device includes at least one of the following: a water meter, an electricity meter, a gas meter, a gas valve, an air conditioner, a television, a refrigerator, curtains, kitchenware, and lamps.

[0019] According to another aspect of the present disclosure, a device information processing apparatus is provided, comprising: a scene determination module, an output module, and a first processing module. The scene determination module is configured to determine a virtual scene associated with a trigger instruction in response to receiving a trigger instruction; wherein the virtual scene includes a virtual space and at least one virtual device in the virtual space, and the at least one virtual device corresponds to at least one real device. The output module is configured to output the virtual scene. The first processing module is configured to process device information of a target real device corresponding to the target virtual device in at least one virtual device in response to receiving a processing instruction from an object for the target virtual device in the at least one virtual device, based on the processing instruction, to obtain a processing result.

[0020] According to another aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein instructions executable by the processor are stored in the memory, and when the instructions are executed by the processor, the processor executes the above method.

[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0022] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided according to the present disclosure when executed by a processor.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0025] FIG1 is a schematic diagram of a system structure according to an embodiment of the present disclosure;

[0026] FIG2 is a schematic flow chart of a method for processing device information according to an embodiment of the present disclosure;

[0027] Figure 3 is a schematic diagram of the connection between the IoT platform and other devices;

[0028] FIG4 is a schematic diagram of the connection between the Internet of Things platform and the third-party platform according to an embodiment of the present disclosure;

[0029] FIG5 is a schematic diagram of the connection between the Internet of Things platform and a third-party application according to an embodiment of the present disclosure;

[0030] FIG6 is a schematic diagram of the connection between the Internet of Things platform and the terminal device according to an embodiment of the present disclosure;

[0031] FIG7 is a schematic diagram of surrounding object photography according to an embodiment of the present disclosure;

[0032] FIG8A is a schematic diagram of a blockchain transaction process according to an embodiment of the present disclosure;

[0033] FIG8B is a schematic diagram of a blockchain network topology according to an embodiment of the present disclosure;

[0034] FIG9 is a schematic diagram of a control principle of a real device according to an embodiment of the present disclosure;

[0035] FIG10 is a schematic diagram showing a user operating a virtual scene according to an embodiment of the present disclosure;

[0036] FIG11 is a schematic structural block diagram of a device information processing apparatus according to an embodiment of the present disclosure; and

[0037] FIG12 is a block diagram of an electronic device suitable for implementing a device information processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure. It should be noted that throughout the drawings, the same elements are represented by the same or similar figure marks. In the following description, some specific embodiments are only for descriptive purposes and should not be understood as any limitation to the present disclosure, but are only examples of the embodiments of the present disclosure. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present disclosure. It should be noted that the shapes and sizes of the components in the figures do not reflect the actual size and proportion, but only illustrate the contents of the embodiments of the present disclosure.

[0039] Unless otherwise defined, technical or scientific terms used in the embodiments of the present disclosure shall have the same general meaning as those skilled in the art. The terms "first," "second," and similar terms used in the embodiments of the present disclosure do not denote any order, quantity, or importance, but are merely used to distinguish different components.

[0040] In some embodiments, users can control and view the operating status of home appliances at home by using a remote control or directly operating the control panel of the home appliance. However, sometimes the user is not at home or there are only elderly people or children living alone who are not familiar with using home appliances. Therefore, it is impossible to effectively control the home appliances and accurately know the operating status of the home appliances.

[0041] In other embodiments, some home appliances provide remote control services, and users can remotely control the home appliances through applications (APPs) for the home appliances. However, on the one hand, such APPs usually display the home appliances as icons or texts on the page, which has poor restoration of real scenes and poor user experience. On the other hand, the current practice is to equip each home appliance with an APP or home appliances of the same brand can share the same APP. In real life, users' home appliances usually involve multiple brands, so users need to install multiple APPs and log in to multiple APPs to control multiple home appliances separately when using them.

[0042] The technical solution provided by the present disclosure maps virtual devices with real devices in real scenes to achieve data interconnection and interoperability. Users can remotely view and control real home appliances in real scenes through virtual devices, thereby facilitating the care of elderly people, children and other people with disabilities living alone, and providing life security for these people. The technical solution provided by the embodiments of the present disclosure can also display three-dimensional virtual space and virtual devices, making the virtual space and virtual devices more vivid and lively, allowing users to control home appliances more intuitively in virtual scenes. In addition, in some embodiments, the embodiments provided by the present disclosure can also connect the device information of multiple home appliances (such as home appliances of multiple brands) to the cloud platform of this system through terminal devices, third-party platforms, third-party applications, etc., so that users can control and view the operating status of multiple home appliances through a single APP, thereby improving the convenience of user operation.

[0043] FIG1 is a schematic diagram of a system structure according to an embodiment of the present disclosure.

[0044] The embodiments of the present disclosure involve home devices 101, sensors 102, image acquisition devices 103, third-party platforms 104, third-party applications 105, network transmission devices, terminal devices 106 and an Internet of Things platform 107.

[0045] The home devices 101 may represent devices whose operating status is to be controlled and queried, for example, the home devices 101 may include water meters, electricity meters, gas meters, gas valves (electric valves are required for control), air conditioners, televisions, refrigerators, washing machines, smart speakers, curtains, kitchen utensils, and lamps. The home devices 101 need to be able to access the network, for example, the home devices 101 access the network through the network layer (NFC / WIFI / 4G / 5G / Bluetooth / ZigBee, etc.), thereby conducting two-way communication with other devices (such as the Internet of Things platform 107 and the third-party platform 104) through the network layer to realize the reporting of device information and the reception of downloaded data.

[0046] Sensor 102 can be used to collect data such as ambient temperature, humidity, and power usage. It should be noted that some home appliances 101 have their own data acquisition devices, which can be used as sensors 102. Additional sensors 102 can also be installed in buildings to collect data based on actual needs.

[0047] The third-party platform 104 may be a platform that provides business data to the Internet of Things platform 107. For example, the third-party platform 104 may include a platform for providing external services to companies such as gas companies and power companies.

[0048] The third-party application 105 is similar to the third-party platform 104 , and the third-party application 105 can provide related services to the IoT platform 107 through an interface.

[0049] The image acquisition device 103 may include a camera, and may be used to capture images of a real space and / or real devices, and then send the images to the IoT platform 107 so that the IoT platform 107 generates a virtual device based on the images.

[0050] Network transmission equipment can include wireless transmission equipment and wireless transmission equipment, such as wired networks, 4G, 5G base stations and other equipment that can transmit data.

[0051] The terminal device 106 is a device that can interact with the user and render a three-dimensional model. The terminal device 106 can interact with the user through, for example, clicks, gesture control, or handle control. The terminal device 106 can include first-category devices and second-category devices. First-category devices can include mobile phones, computers, etc. Second-category devices can include AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) devices, such as VR glasses.

[0052] The IoT platform 107 can be used for data processing, data storage, and instruction processing, thereby enabling data interconnection and interoperability between various devices. In practical applications, the IoT platform 107 can include a single server or a server cluster, which can include multiple servers such as business servers, storage servers, and other servers. The IoT platform 107 can connect to the third-party platform 104 to access business data. The IoT platform 107 can also store accessed data. The IoT platform 107 can also perform data processing and shared transmission.

[0053] The IoT platform 107 can be pre-connected with the third-party platform 104, the third-party application 105, the terminal device 106, etc., thereby obtaining relevant data from the third-party platform 104, the third-party application 105, and the terminal device 106, and storing the relevant data in a database. The relevant data may include, for example, user information and device information of the home device 101. The device information may include operating status information (such as air conditioning temperature, power consumption, remaining power, etc.), attribute information (such as device name, device identification, device model, etc.), etc. The connection between the IoT platform 107 and the third-party platform 104, the third-party application 105, and the terminal device 106 will be described below and will not be repeated here.

[0054] In addition, the three-dimensional model can be generated by pre-building the scene, rendering, and modeling using Unreal Engine or other three-dimensional production tools. The three-dimensional model can be packaged and generated in the software application of the terminal device 106. When the user operates the terminal device 106, the terminal device 106 can respond to the trigger instruction generated based on the operation, and then load the three-dimensional model. Alternatively, the three-dimensional model can be stored in the Internet of Things platform 107. When the user operates the terminal device 106, the terminal device 106 sends the trigger instruction to the Internet of Things platform 107. The Internet of Things platform 107 sends the three-dimensional model to the terminal device 106, and then the terminal device 106 loads the three-dimensional model. In some embodiments, the three-dimensional model represents a virtual scene. The virtual scene may include a virtual space and virtual devices, wherein the virtual space corresponds to the real space, and the virtual devices correspond one-to-one with the real devices. In addition, at least a portion of the real devices are in the real space, and correspondingly, at least a portion of the virtual devices are in the virtual space.

[0055] FIG2 is a schematic flowchart of a method for processing device information according to an embodiment of the present disclosure.

[0056] As shown in FIG. 2 , the present disclosure provides a method 200 for running an application program. The method 200 may include operations S210 to S230 .

[0057] In operation S210 , in response to receiving a trigger instruction, a virtual scene associated with the trigger instruction is determined; wherein the virtual scene includes a virtual space and at least one virtual device in the virtual space, and the at least one virtual device corresponds to at least one real device.

[0058] In operation S220 , the virtual scene is output.

[0059] In operation S230 , in response to receiving a processing instruction for a target virtual device in at least one virtual device from the object, device information of the target real device corresponding to the target virtual device in at least one real device is processed based on the processing instruction to obtain a processing result.

[0060] For example, the object can be a user, and the user can log in to his or her account on the terminal device, and then the terminal device can display the initial virtual space and initial virtual devices. The user can then perform operations on the terminal device, for example, the user can select and modify the virtual space on the terminal device, and can add, modify, and delete virtual devices in the virtual space, and then save the user's operation results locally. When the account is logged in next time, the virtual scene can be determined based on the last saved operation result, and the terminal device can render the virtual scene so that the user can perform subsequent processing based on the virtual scene. Virtual devices correspond one-to-one to real devices, and real devices can be, for example, the home appliances mentioned above.

[0061] For another example, after obtaining an operation result, the operation result can be stored on the IoT platform, and an association relationship can be established between the virtual scene and the user identifier. The association between the user identifier and the virtual scene can indicate that the virtual scene was generated based on the user's operation. The next time the account is logged in, the login operation can generate a trigger instruction, and the user identifier in the trigger instruction can be sent by the terminal device to the IoT platform. The IoT platform can query the virtual scene after the user's last operation based on the account, and send the virtual scene to the terminal device. The terminal device can then render the virtual scene so that the user can perform subsequent processing based on the virtual scene.

[0062] For example, after a terminal device renders a virtual scene, the user can operate a target virtual device in the virtual scene. This operation can include querying the device status and controlling the device status. This operation triggers a processing instruction, which the terminal device can then send to the IoT platform. The IoT platform returns the queried device information to the terminal device or sends a control instruction to the real device corresponding to the target virtual device to control the operating status of the real device. It should be noted that the device information of the virtual device in the virtual scene can be determined based on the user's logged-in account.

[0063] By creating virtual spaces and virtual devices using the technical solutions provided by the embodiments of this disclosure, users can view relevant information about real devices within the virtual scene and control their operating status, thus resolving the issue of being unable to control home appliances both locally and remotely. Furthermore, restoring the three-dimensional space and devices within the metaverse scene makes the virtual space and virtual devices more realistic and vivid.

[0064] Figure 3 is a schematic diagram of the connection between the IoT platform and other devices.

[0065] In this embodiment, the Internet of Things platform 302 can be connected with other devices (such as a third-party platform 304, a third-party application 305, and a terminal device 306) to obtain relevant data, such as user information related to the user, device information related to the real device, and the correspondence between user information and device information. The device information includes, for example, the current operating status of the device, device identification, historical setting parameters, etc. The Internet of Things platform 302 connects with other devices to obtain relevant data, which can provide a basis for subsequent querying and controlling the device information of the real device, ensuring the smooth progress of subsequent processing. The process of connecting the Internet of Things platform 302 with the third-party platform 304, the third-party application 305, and the terminal device 306 is described in detail below and will not be repeated here.

[0066] Furthermore, an initial virtual scene can be pre-generated and stored in the IoT platform 302 or terminal device 303. Users can use the terminal device 303 to manipulate the initial virtual scene, and the IoT platform 302 can update the virtual space and virtual devices in the initial virtual scene based on the user's manipulations. The IoT platform 302 can then process the virtual devices in the virtual scene based on data obtained from other devices. For example, this processing can include viewing device information on real devices or controlling the operating status of real devices.

[0067] The following describes the connection between the IoT platform and third-party platforms, third-party applications, and terminal devices in conjunction with embodiments.

[0068] FIG4 is a schematic diagram of the connection between the Internet of Things platform and the third-party platform according to an embodiment of the present disclosure.

[0069] As shown in Figure 4, this embodiment illustrates the connection between an IoT platform 401 and a third-party platform. Examples of third-party platforms include public cloud platforms 402 and 403, and third-party systems 404. Each third-party platform may include multiple functional modules to implement its own logic. For example, a third-party platform may include middleware, application systems, databases, platform access, APIs, authentication systems, business systems, and user management systems. Each functional module may be implemented by software and / or hardware, and this embodiment does not limit the third-party platform.

[0070] Information can be transmitted between the third-party platform and the IoT platform 401. For example, the IoT platform 401 sends a request to the third-party platform, and the third-party platform responds to the IoT platform 401. The IoT platform 401 pre-agreed with the third-party platform on request parameters, response parameters, and format. This embodiment does not limit the request parameters, response parameters, and format.

[0071] In an example, the request parameters are shown in Table 1, and the response parameters are shown in Table 2, wherein the business data is returned through the data field, and both the request parameters and the response parameters can be in JSON format.

[0072] Table 1 Request parameters

[0073] Table 2 Response parameters

[0074] FIG5 is a schematic diagram of the connection between the Internet of Things platform and a third-party application according to an embodiment of the present disclosure.

[0075] As shown in Figure 5, this embodiment illustrates the connection between an IoT platform 501 and a third-party application. The third-party platform includes, for example, applications 502, 503, 504, and 505, each of which can provide multiple interfaces. The IoT platform can also obtain authorization for some of the interfaces of applications 502 and 503, allowing it to call certain interfaces of those applications. The IoT platform can also obtain full authorization for some applications 504 and 505, allowing it to call all interfaces of those applications. IoT platform 501 transfers data between applications by calling their interfaces.

[0076] FIG6 is a schematic diagram of the connection between the Internet of Things platform and the terminal device according to an embodiment of the present disclosure.

[0077] As shown in Figure 6, this embodiment illustrates the connection between the IoT platform and terminal devices. For example, terminal devices 602, 603, and 604 can directly access the IoT platform 601 via a wireless network. The multiple terminal devices 602, 603, and 604 can have different operating systems. For example, the three operating systems in Figure 6 can include Android, Linux, and Windows.

[0078] For example, a terminal device includes a software application app and DMA (Direct Memory Access). The app may include an IoT (Internet of Things) SDK (Software Development Kit). The DMA running on the terminal device allows direct interaction with the software application app. The software application app leverages the DMA's capabilities to report data generated by the terminal device to the IoT platform. The DMA also acts as an agent, receiving instructions from the IoT platform and executing related operations.

[0079] After the IoT platform is connected with third-party platforms, third-party applications, and terminal devices, the IoT platform can obtain the identification information, device information, and mapping information between the real devices. The mapping information represents the correspondence between the identification information and device information of the real devices. The IoT platform can also store this information in the database.

[0080] After the user logs in to his / her personal account in the terminal device, a virtual scene can be generated based on the user's operation. The following describes the process of generating the virtual scene in conjunction with the embodiment.

[0081] For example, a 3D model can be generated by pre-building scenes, rendering, and modeling using Unreal Engine or other 3D production tools. This 3D model can represent the initial virtual space and initial 3D virtual devices. Modeling engineers can manually model the initial virtual space and devices and store them in the Unreal Engine as data packages. Modeling engineers can then define product logic based on actual needs. This logic can be used to control operations such as displaying and hiding virtual devices, changing their positions, and switching scenes.

[0082] In one example, a user can use an initial virtual scene to generate a virtual scene. For example, after a user registers and logs in to an account for the first time, the terminal device can load and display the initial virtual scene. The initial virtual scene can include an initial virtual space and / or initial virtual devices. For example, a virtual space of a predetermined apartment type is displayed by default, and a virtual air conditioner and virtual lamps are displayed in predetermined locations in the virtual space.

[0083] Next, for example, the user may directly select a confirmation option, thereby determining the initial virtual scene as the aforementioned virtual scene.

[0084] For another example, a user can trigger a modification instruction by clicking a button, generating a predetermined gesture, or other operational behavior. The modification instruction may include a space modification instruction and / or a device modification instruction. The modification instruction may include a modification method, which may include deleting part of the virtual scene, moving part of the virtual scene, deleting at least one initial virtual device, modifying the display position of the initial virtual device in the virtual scene, modifying the virtual image of the virtual device (for example, modifying the color, size, material, etc.), modifying the attribute information of the initial virtual device, etc. Taking the space modification instruction as an example, the terminal device can directly modify the initial virtual space based on the modification instruction. Or the terminal device sends the space modification instruction to the Internet of Things platform, and the Internet of Things platform modifies and maintains the initial virtual space, and also returns parameters to the terminal device so that the terminal device renders the modified virtual space, thereby updating the virtual space. The device modification instruction is similar to the space modification instruction and will not be repeated here. In this embodiment, the user can modify the virtual space and virtual device according to actual needs so that the virtual scene is consistent with the real scene.

[0085] In another example, users can generate virtual scenes using a resource library. For example, the resource library includes candidate virtual spaces and candidate virtual devices. Clicking on the resource library displays multiple candidate virtual devices in the library. Next, the user's actions generate an add instruction, which includes the identification information and target location information of the candidate virtual device. Based on the identification information, the terminal device can call the corresponding virtual device and display it at the target location in the virtual space. In this embodiment, users can add virtual devices based on their actual needs, increasing operational flexibility.

[0086] In another example, for virtual devices already in a virtual space, users can perform actions based on their actual needs to trigger device processing instructions. Based on these instructions, the modified virtual scene can then be modified multiple times, similar to the above-described modification methods. For example, to delete a virtual device, the user clicks the virtual device to be deleted in the virtual scene and selects the Delete button. This action triggers a processing instruction, which the terminal device sends to the IoT platform. After receiving the processing instruction, the IoT platform updates the status of the relevant fields in the database and returns a response message, deleting the virtual device. In practice, deleting a virtual device can be achieved by adjusting the display effect to hidden. This process only processes the virtual device in the current scene and does not affect the initial virtual scene. Moving a virtual device is similar to deleting a virtual device, except that deleting a virtual device involves adjusting the display effect to hidden, while modifying the virtual device's position changes the virtual device's coordinates in the virtual space and synchronously saves the updated coordinates on the IoT platform. In this embodiment, users can modify virtual devices based on their actual needs, increasing operational flexibility.

[0087] In another example, a user can generate a virtual scene by taking an image of a real scene. For example, a user uses a terminal or other device with an image acquisition function (such as a depth camera) to take a picture of a real space, in which at least one real device is placed, so that the multiple real space images include a device image for at least one real device. Then, a first predetermined image processing algorithm is used to process the multiple real images. For example, the first predetermined image processing algorithm may include a target detection model, and the target detection model detects the category of at least one real device in the multiple real space images. Then, based on the category of at least one real device, a virtual device of the category can be selected from the material library, and the selected virtual image can be added to the virtual space. In this embodiment, the user can automatically generate a virtual scene by taking an image of a real scene, which can reduce the amount of manual operations for the user to select and adjust the virtual space and virtual devices, thereby improving the user's convenience and user experience.

[0088] In another example, the user can generate a virtual scene by capturing an image of a real device. For example, the user uses a terminal or other device with an image acquisition function to capture an image of the real device.

[0089] For example, taking the terminal device as a camera capable of collecting depth information, after the camera starts the shooting program, the user can register and log in to a personal account, then start the modeling module and turn on the camera to shoot. As shown in Figure 7, two auxiliary lines 701 and 702 can be displayed in the three-dimensional world in the camera. The two auxiliary lines divide the surface of the real device into three sub-areas from top to bottom or other predetermined directions. The user can perform circumstantial photography of each sub-area to obtain 12 or other images of each sub-area. If the user completes the shooting, he can click the "Done" option, after which the camera can send multiple images to the Internet of Things platform. The Internet of Things platform uses a second predetermined image processing algorithm to process multiple images of multiple sub-areas, thereby generating a virtual device corresponding to the real device. The Internet of Things platform then adds the virtual device to the material library so that the user can add the virtual device to the virtual scene based on the material library. In this embodiment, if the material library lacks virtual devices corresponding to certain real devices, the user can generate virtual devices by shooting real devices, thereby improving the consistency between the virtual scene and the real scene. In addition, circumstantial photography can accurately obtain a complete image of the real device, thereby ensuring the accuracy of the generated virtual device. It is understandable that in other embodiments, the auxiliary line may not be displayed and the number of sub-areas may be one.

[0090] For example, the second predetermined image processing algorithm may include a three-dimensional stitching algorithm or a NeRF (Neural Radiance Fields) model. Taking the NeRF model as an example, the NeRF model is implemented based on a neural network and requires pre-training. This embodiment does not limit the training process. For example, multiple images of a real object, the camera's position and orientation, and other information can be input into the NeRF model. The NeRF model performs image feature extraction, sampling, position encoding, and other processing. The NeRF model then learns the neural radiation field function for each point in the scene and uses these functions to predict images from any perspective. After training is complete, the multiple images and other information are input into the NeRF model. The NeRF model can output the color and depth values ​​of each point in the scene and render it, thereby generating a virtual device. The virtual device is then added to the resource library. The user can also add relevant data for the virtual device, which may include information such as a device identification code and room number. Based on user operations, the IoT platform can send the virtual device to the terminal device for display.

[0091] Furthermore, in some embodiments, blockchain technology can be used to implement transactions for real devices. For example, a financial payment system's blockchain can be connected to a terminal device via an SDK or API interface, ensuring transaction security. This disclosure maps real devices into virtual devices and facilitates user transactions for real devices through blockchain transactions, further enhancing user convenience. Furthermore, using the financial payment system's blockchain ensures transparency and queryability of each transaction. A schematic diagram of the blockchain transaction process and a schematic diagram of the blockchain network topology are shown in Figures 8A and 8B. This embodiment does not limit the implementation of the blockchain.

[0092] The above embodiments describe the connection between the IoT platform and third-party platforms, third-party applications, and terminal devices. After the connection, the IoT platform can obtain relevant information from the third-party platforms, third-party applications, and terminal devices. In addition, the above embodiments also describe the process of generating a virtual scene based on user operations. In this process, the user identifier and device identifier can be bound. Then, after the user logs back in and enters the display page, the IoT platform can obtain the user information under the account and the device information of the real device from the database based on the user identifier in the account, such as the user name, the "My Home" list under the user name, and the device information of the real devices in the home.

[0093] FIG9 is a schematic diagram illustrating a method for controlling a real device according to an embodiment of the present disclosure.

[0094] As shown in Figure 9, after the above processing, the terminal device can display the virtual scene, and the virtual devices 903 in the virtual scene are mapped one by one to the real devices 902 in the real scene. Next, after the IoT platform 901 obtains relevant information and generates the virtual scene, the user can process the device information of the real device 902 based on the virtual device 903 in the virtual scene according to actual needs. For example, through user operation, a control signal is generated to enable the IoT platform to control the real device 902. The relevant information of the controlled real device 902 is then returned to the virtual device 903 as a reference for display.

[0095] The following describes the process of processing device information in conjunction with an embodiment.

[0096] In one embodiment, the processing includes controlling the running state of the device, and accordingly, the processing instruction includes a control instruction. In this embodiment, the user can remotely control the running state of the real device to meet the user's control needs.

[0097] For example, after the user registers and logs in to the account, the terminal device can load the three-dimensional model of the virtual scene. After entering the virtual scene, the user can view the virtual scene from the first perspective or the third perspective. The virtual scene includes a virtual space and multiple virtual devices, and the multiple virtual devices are arranged in the virtual space. Taking the terminal device as the second type of device mentioned above (such as VR glasses) as an example, the user can select the target virtual device from multiple virtual devices by clicking or generating gestures. At this time, a control panel pops up in the virtual scene. The control panel can be implemented by a UMG plug-in or a custom package plug-in. The user clicks the start button, and the terminal device sends a control instruction to the real device corresponding to the target virtual device, or the terminal device outputs the "start" field to the Internet of Things platform, and the Internet of Things platform sends a control instruction to the real device corresponding to the target virtual device to control the real device to start up. The process of the user clicking other buttons to trigger the control instruction is similar to clicking the start button. The difference is that the transmitted fields are different.

[0098] In one embodiment, the processing includes querying device information, and accordingly, the processing instruction includes a query instruction. In this embodiment, the user can remotely query the device information of the real device, meeting the user's query needs.

[0099] For example, after a user registers and logs in to an account, the terminal device can load a three-dimensional model of a virtual scene. After entering the virtual scene, the user can view the virtual scene from either a first-person or third-person perspective. The virtual scene includes a virtual space and multiple virtual devices arranged within the virtual space. Taking the terminal device as the second type of device (e.g., VR glasses) as an example, the user can click a query option on a target virtual device among the multiple virtual devices. The query option is used to query the status of the real device associated with the target virtual device, such as its on / off status and operating parameters (air conditioning temperature, operating mode, etc.). Clicking the query option generates a query instruction, which the terminal device sends to the IoT platform. The query instruction includes the fields to be queried. The IoT platform queries the database for data based on information such as the user ID, device ID, and the fields to be queried, and then returns the query results to the terminal device. A location for displaying the query field is preconfigured on the target virtual device, and the terminal device displays the query results at this location after receiving them. For example, if the target virtual device is an air conditioner, this location can display information such as the current air conditioning temperature and indoor temperature.

[0100] In another embodiment, the process includes recharging and paying for a real device. Similar to the above process, the user clicks or performs a predetermined gesture on the virtual device and then selects the recharge and payment button. The terminal device or IoT platform then invokes a related interface based on this action and inputs relevant fields such as user identification, device identification, and recharge information into the interface, completing the recharge and payment process for the real device.

[0101] The above embodiment is illustrated using the terminal device as a second-category device as an example. If the terminal device is the above-mentioned first-category device (such as a mobile phone), the user needs to click on the target virtual device in the virtual scene through the screen, and the rest of the data transmission process is the same as that of the second-category device.

[0102] FIG10 is a schematic diagram illustrating a user's operation on a virtual scene according to an embodiment of the present disclosure.

[0103] For example, a new user first connects to the network, completes user registration and privacy authorization, and enters information such as address, building, and house number. The user can then generate a virtual scene based on the initial virtual scene or a captured image. The IoT platform stores the relevant information about the virtual scene. The same user can be associated with multiple virtual scenes, such as "My Home 1," "My Home 2," and "My Home 3" in the figure. The user selects one of the virtual scenes, such as "My Home 1." The terminal device then pulls the relevant information about the virtual scene from the IoT platform and renders it. The user can then perform operations within the virtual scene, such as querying device information or controlling a real device. The terminal device or IoT platform then sends commands to the real device based on the user's operations to control the real device until the user exits the operation.

[0104] FIG11 is a schematic structural block diagram of a device information processing apparatus according to an embodiment of the present disclosure.

[0105] As shown in FIG. 11 , the device information processing apparatus includes: a scene determination module 1110 , an output module 1120 and a first processing module 1130 .

[0106] The scene determination module 1110 is used to determine a virtual scene associated with the trigger instruction in response to receiving the trigger instruction; wherein the virtual scene includes a virtual space and at least one virtual device in the virtual space, and the at least one virtual device corresponds to at least one real device.

[0107] The output module 1120 is used to output the virtual scene.

[0108] The first processing module 1130 is configured to, in response to receiving a processing instruction for a target virtual device in at least one virtual device from the object, process device information of the target real device corresponding to the target virtual device in at least one real device based on the processing instruction to obtain a processing result.

[0109] In this embodiment, the processing instruction includes a query instruction; the first processing module includes: a query submodule, which is used to query device information for the target virtual device from the database based on the identification information in the query instruction; the device information of the target virtual device represents the status of the real device related to the target virtual device.

[0110] In this embodiment, the apparatus further includes an acquisition module and a storage module. The acquisition module is configured to acquire identification information and device information of a real device based on at least one of a third-party platform, a third-party application, a terminal device, and a real device. The storage module is configured to store the identification information, device information, and mapping information in a database; the mapping information represents the correspondence between the identification information and device information of the real device.

[0111] In this embodiment, the processing instruction includes a control instruction; the first processing module includes: a control submodule, which is used to control the operating state of the target real device based on the control instruction.

[0112] In this embodiment, the scene determination module includes: a scene determination submodule, which is used to determine a virtual scene associated with the object identifier based on the object identifier in the trigger instruction.

[0113] In this embodiment, the apparatus further includes: a first image acquisition module, a category determination module, a first device determination module, and a first adding module. The first image acquisition module is configured to acquire multiple real-space images obtained by performing image acquisition of a real space; wherein the real space includes at least one real device, and the multiple real-space images include a device image for the at least one real device. The category determination module is configured to determine the category of the at least one real device based on the device image of the at least one real device. The first device determination module is configured to determine a virtual device corresponding to the at least one real device from a material library based on the category of the at least one real device. The first adding module is configured to add the virtual device determined from the material library to the virtual space.

[0114] In this embodiment, the apparatus further includes: a second image acquisition module, a generation module, and a second addition module. The second image acquisition module is configured to acquire multiple device images obtained by capturing images of real devices. The generation module is configured to generate virtual devices corresponding to the real devices based on the multiple device images. The second addition module is configured to add the generated virtual devices to the material library.

[0115] In this embodiment, the real device includes at least one device sub-area distributed along a predetermined direction, and the multiple device images are obtained by performing ambient photography on each device sub-area.

[0116] In this embodiment, the apparatus further includes a space modification module and a space determination module. The space modification module is configured to, in response to receiving a space modification instruction from an object for the initial virtual space, modify the initial virtual space based on the space modification instruction. The space determination module is configured to determine the modified initial virtual space as the virtual space.

[0117] In this embodiment, the apparatus further includes a device modification module and a second device determination module. The device modification module is configured to, in response to receiving a device modification instruction from a subject for at least one initial virtual device, modify the at least one initial virtual device based on the device modification instruction. The second device determination module is configured to determine the modified initial virtual device as the virtual device.

[0118] In this embodiment, the apparatus further includes: a third adding module configured to, in response to receiving an adding instruction from the object for at least one candidate virtual device in the material library, add the at least one candidate virtual device as a virtual device to the virtual space.

[0119] In this embodiment, the above-mentioned device also includes: a second processing module, which is used to process the virtual device in the virtual space in response to receiving a device processing instruction from the object based on the processing mode in the device processing instruction; wherein the processing mode includes at least one of the following: deleting the virtual device, modifying the virtual image of the virtual device, and modifying the attribute information of the virtual device.

[0120] In this embodiment, the virtual space corresponds to the real space, and at least a portion of the at least one real device is located in the real space.

[0121] In this embodiment, the real device includes at least one of the following: a water meter, an electricity meter, a gas meter, a gas valve, an air conditioner, a television, a refrigerator, curtains, kitchenware, and lamps.

[0122] FIG12 is a block diagram of an electronic device suitable for implementing a device information processing method according to an embodiment of the present disclosure.

[0123] As shown in Figure 12, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1208 into a random access memory (RAM) 1203. The processor 1201 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include an onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0124] Various programs and data required for the operation of the electronic device 1200 are stored in the RAM 1203. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. The processor 1201 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1202 and / or the RAM 1203. It should be noted that the programs may also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0125] According to an embodiment of the present disclosure, electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to bus 1204. Electronic device 1200 may further include one or more of the following components connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a LAN card or a modem. Communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in drive 1210 as needed, so that computer programs read from the removable media can be installed into storage section 1208 as needed.

[0126] The present disclosure also provides a non-transitory computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0127] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.

[0128] Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the method for verifying process data of a product provided by the embodiments of the present disclosure.

[0129] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 1201 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0130] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1209, and / or installed from the removable medium 1211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0131] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209 and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0132] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0135] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0136] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A device information processing method, comprising: In response to receiving a trigger instruction, determining a virtual scene associated with the trigger instruction; wherein the virtual scene includes a virtual space and at least one virtual device in the virtual space, and the at least one virtual device corresponds to at least one real device; outputting the virtual scene; and In response to receiving a processing instruction from an object for a target virtual device in the at least one virtual device, based on the processing instruction, device information of a target real device in the at least one real device corresponding to the target virtual device is processed to obtain a processing result.

2. The method according to claim 1, wherein: The processing instruction includes a query instruction; and the processing of device information of a target real device corresponding to the target virtual device in the at least one real device includes: Based on the identification information in the query instruction, device information for the target virtual device is queried from a database; the device information of the target virtual device represents the state of a real device related to the target virtual device.

3. The method according to claim 2, further comprising: Based on at least one of a third-party platform, a third-party application, a terminal device, and a real device, obtaining identification information and device information of the real device; and storing the identification information, device information and mapping information in the database; The mapping information represents the corresponding relationship between the identification information of the real device and the device information.

4. The method according to claim 1, wherein: The processing instruction includes a control instruction; and the processing of device information of a target real device corresponding to the target virtual device in the at least one real device includes: Based on the control instruction, the operating state of the target real device is controlled.

5. The method according to claim 1, wherein: Determining the virtual scene associated with the trigger instruction includes: Based on the object identifier in the trigger instruction, the virtual scene associated with the object identifier is determined.

6. The method according to any one of claims 1 to 5, further comprising: Acquire a plurality of real space images obtained by performing image acquisition on the real space; wherein the real space includes at least one real device, and the plurality of real space images include a device image for the at least one real device; determining a category of the at least one real device based on the device image of the at least one real device; and Based on the category of the at least one real device, determining a virtual device corresponding to the at least one real device from a material library; Adding a virtual device determined from the material library to the virtual space.

7. The method according to any one of claims 1 to 6, further comprising: Acquire multiple device images obtained by performing image acquisition on real devices; Based on the multiple device images, generating a virtual device corresponding to the real device; as well as The generated virtual device is added to the material library.

8. The method according to claim 7, wherein: The real device includes at least one device sub-area distributed along a predetermined direction, and the multiple device images are obtained by performing ambient photography on each device sub-area.

9. The method according to claim 1, further comprising: In response to receiving a space modification instruction for the initial virtual space from the object, modifying the initial virtual space based on the space modification instruction; as well as The modified initial virtual space is determined as the virtual space.

10. The method according to claim 1, further comprising: In response to receiving a device modification instruction for at least one initial virtual device from the object, modifying at least one initial virtual device based on the device modification instruction; as well as The modified initial virtual device is determined as the virtual device.

11. The method according to claim 1, further comprising: In response to receiving an adding instruction from the object for at least one candidate virtual device in the material library, the at least one candidate virtual device is taken as a virtual device and added to the virtual space.

12. The method according to claim 1, further comprising: In response to receiving a device processing instruction from the object, processing a virtual device in the virtual space based on a processing mode in the device processing instruction; The processing mode includes at least one of the following: deleting the virtual device, modifying the virtual image of the virtual device, and modifying the attribute information of the virtual device.

13. The method according to claim 1, wherein: The virtual space corresponds to a real space, and at least a portion of the at least one real device is located in the real space.

14. The method according to claim 1, wherein: The real device includes at least one of the following: a water meter, an electricity meter, a gas meter, a gas valve, an air conditioner, a television, a refrigerator, a curtain, kitchenware, and a lamp.

15. A device information processing apparatus, comprising: A scene determination module, configured to determine a virtual scene associated with the trigger instruction in response to receiving the trigger instruction; wherein the virtual scene includes a virtual space and at least one virtual device in the virtual space, and the at least one virtual device corresponds to at least one real device; Output, used to output the virtual scene; and The first processing module is used to respond to receiving a processing instruction for a target virtual device in the at least one virtual device from the object, and based on the processing instruction, process device information of the target real device corresponding to the target virtual device in the at least one real device to obtain a processing result.

16. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 14.

17. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images