Environmental model generation method, display method, device, equipment and storage medium

Abstract

The present application provides a method for generating an environmental model, a display method, an apparatus, an electronic device, and a storage medium, and relates to the field of Internet of Things technology. The generation method includes: displaying a spatial layout diagram corresponding to a target scene; the spatial layout diagram is generated based on the relationship between the trajectory of a first target object in the target scene and each space in the target scene; identifying each second target object deployed in the target scene, and obtaining the position of each identified second target object in the target scene; generating an environmental model of the target scene based on the position of each second target object in the target scene and the spatial layout diagram corresponding to the target scene, for describing the status of the first target object and / or the second target object in the target scene. The present application associates the second target object with the spatial layout diagram, and provides a device monitoring method based on spatial positioning, which can intuitively monitor various equipment and facilities in the target scene, thereby helping to improve the effectiveness of the device control service.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things technology. Specifically, the present application relates to an environment model generation method, display method, device, electronic device and storage medium. Background Art

[0002] With the development of the field of Internet of Things technology, smart devices can be deployed in various scenarios. For example, the scenario can be a smart home scenario, so as to provide people with device control services for each smart device in the scenario, so that people can enjoy a more comfortable living / learning / working environment. This also makes people increasingly concerned about how to better monitor the smart devices in the scene so as to adjust the environment in a timely manner.

[0003] In the current device monitoring solution, users cannot intuitively monitor the various smart devices in the smart home scene. For example, there are multiple air conditioners deployed in the living room, and the temperature in the upper left corner of the living room is relatively high. Since the user is not clear about the location of each air conditioner in the living room, he cannot determine which air conditioner should be controlled to achieve the purpose of adjusting the temperature in the upper left corner of the living room, thereby affecting the effectiveness of the device control service.

[0004] As can be seen from the above, the relevant technology still lacks a solution for intuitively monitoring each smart device in the scene, which in turn affects the effectiveness of the device control service. Summary of the Invention

[0005] This application provides a method for generating and displaying an environmental model, a device, an electronic device, and a storage medium, which can solve the problem in related technologies that the lack of an intuitive device monitoring solution affects the effectiveness of device control services. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for generating an environmental model includes: displaying a spatial layout diagram corresponding to a target scene; the spatial layout diagram is generated based on the relationship between the trajectory of a first target object in the target scene and each space in the target scene; identifying each second target object deployed in the target scene, and obtaining the position of each identified second target object in the target scene; generating an environmental model of the target scene based on the position of each second target object in the target scene and the spatial layout diagram corresponding to the target scene; the environmental model is used to describe the state of the first target object and / or second target object in the target scene.

[0007] According to one aspect of the present application, a method for displaying an environmental model includes: displaying an environmental model corresponding to a target scene, the environmental model including the status of each first target object and / or each second target object in the target scene; the environmental model is generated based on a spatial layout diagram of the target scene and the positions of each second target object in the target scene, the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and each space in the target scene; if it is identified that the status of each first target object and / or each second target object has changed, the updated status of the first target object and / or second target object is displayed in the environmental model.

[0008] According to one aspect of the present application, a device for generating an environmental model includes: a layout diagram display module for displaying a spatial layout diagram corresponding to a target scene; the spatial layout diagram is generated based on the relationship between the trajectory of a first target object in the target scene and each space in the target scene; an object recognition module for identifying each second target object deployed in the target scene and obtaining the position of each identified second target object in the target scene; a model generation module for generating an environmental model of the target scene based on the position of each second target object in the target scene and the spatial layout diagram corresponding to the target scene; the environmental model is used to describe the state of the first target object and / or second target object in the target scene.

[0009] In an exemplary embodiment, the model generation module includes: an area determination unit, used to determine the corresponding area of each second target object in the spatial layout diagram based on the position of each second target object in the target scene; a mark display unit, used to display the object mark of each second target object in the corresponding area in the spatial layout diagram to generate an environmental model of the target scene; the object mark of the second target object is used to indicate the status of the second target object.

[0010] In an exemplary embodiment, the device also includes: an object re-identification module, which is used to identify the second target object newly deployed in the target scene based on the correlation between the signal characteristics of each displayed second target object and the echo signal; the echo signal is obtained by the radar equipment capturing the micro-motion signal generated when the second target object is working; and a model updating module, which is used to display the object mark of the newly deployed second target object in the corresponding area of the spatial layout diagram, and update the environmental model of the target scene.

[0011] In an exemplary embodiment, the second target object includes an electrically controlled object; the electrically controlled object refers to a device that can generate a micro-motion signal when in operation; the object recognition module may further include: a first positioning unit, configured to locate the position of the electrically controlled object based on the echo signal, identify the position of the electrically controlled object in the target scene, and determine the corresponding area of the electrically controlled object in the spatial layout diagram; wherein the echo signal is obtained by a radar device capturing the micro-motion signal generated by the electrically controlled object when in operation.

[0012] In an exemplary embodiment, the object recognition module may further include: a state recognition unit for identifying that the state of the electronically controlled object is on based on the capture of the micro-motion signal generated by the radar device when the electronically controlled object is working, so as to mark the electronically controlled object in the on state in the corresponding area determined by the spatial layout diagram.

[0013] In an exemplary embodiment, the second target object includes a non-electrically controlled object; the non-electrically controlled object refers to a facility that cannot generate a micro-motion signal; the object recognition module may also include: a second unit unit, used to locate the position of the first target object interacting with the non-electrically controlled object based on the echo signal, and obtain the position of the first target object in the target scene; the echo signal is obtained by reflecting the transmission signal of the radar equipment through the first target object; a position recognition unit, used to identify the position of the non-electrically controlled object in the target scene according to the position of the first target object in the target scene, so as to determine the corresponding area of the non-electrically controlled object in the spatial layout diagram.

[0014] In an exemplary embodiment, the second target object includes a smart device, and the object recognition module may further include: a range determination unit, configured to determine the area range of the smart device in the target scene if it is detected that the state of the smart device changes from an off state to an on state; and a first type recognition unit, configured to identify the device type of the smart device as the object type of the second target object if the radar device captures a micro-motion signal generated when the second target object is working within the determined area range, so as to mark the second target object according to the device type of the smart device in the spatial layout diagram.

[0015] In an exemplary embodiment, the object recognition module may further include: a second type recognition unit, configured to identify the object type of the second target object in response to a type marking operation on the second target object; or to identify the object type of the second target object in response to a recognition event triggered by an image acquisition device on the second target object.

[0016] In an exemplary embodiment, the device may also include: a trajectory information acquisition module, used to obtain the trajectory information of the first target object in each space in the target scene, draw and display the trajectory map of each space; the trajectory information is used to indicate the movement trajectory of the first target object in each space; an entrance and exit marking module, used to mark the entrance and exit positions of each space on the trajectory map of each space; a layout map acquisition module, used to obtain the space layout map by matching the trajectory map of each space with the marked entrance and exit positions.

[0017] In an exemplary embodiment, the trajectory information acquisition module includes: a trajectory tracking unit, which is used to track the trajectory of the first target object based on the echo signal to obtain the trajectory information of the first target object in each space; the echo signal is obtained by the radar equipment's transmission signal reflected by the first target object.

[0018] In an exemplary embodiment, the entrance and exit marking module includes: a position determining unit for determining, in response to a space opening and closing event triggered in each space, the position of the first target object in each space when the space opening and closing event is triggered.

[0019] In an exemplary embodiment, the entrance and exit marking module includes: an adjacent space determination unit, used to determine two adjacent spaces associated with the connected position in response to a marking operation on the connected position; the connected position is used to indicate the position of the first target object when entering and exiting between the two adjacent spaces.

[0020] According to one aspect of the present application, an environment model display device includes: a model display module for displaying an environment model corresponding to a target scene; the environment model includes the status of each first target object and / or each second target object in the target scene; the environment model is generated based on the spatial layout diagram of the target scene and the position of each second target object in the target scene; the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and each space in the target scene; a change identification module for displaying the updated status of the first target object and / or the second target object in the environment model if it is identified that the status of each first target object and / or each second target object has changed.

[0021] According to one aspect of the present application, an electronic device includes: at least one processor and at least one memory, wherein a computer program is stored in the memory, and the processor reads the computer program in the memory; when the computer program is executed by the processor, the method described above is implemented.

[0022] According to one aspect of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by a processor, the method described above is implemented.

[0023] According to one aspect of the present application, a computer program product includes a computer program, the computer program is stored in a storage medium, a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, so that the computer device implements the method described above when executing the computer program.

[0024] The beneficial effects of the technical solution provided by this application are:

[0025] In the above technical solution, a spatial layout diagram corresponding to the target scene is generated according to the relationship between the trajectory of the first target object in the target scene and the various spaces in the target scene, and each second target object deployed in the target scene is identified to obtain the position of each second target object in the target scene. Then, based on the spatial layout diagram corresponding to the target scene and the position of each second target object in the target scene, an environmental model of the target scene is generated. The environmental model can be used to describe the status of each second target object in the target scene. Thus, each second target object in the target scene is associated with the spatial layout diagram corresponding to the target scene through the environmental model, providing a device monitoring solution based on spatial positioning, which can monitor each second target object (i.e., various equipment and facilities) in the target scene intuitively, dynamically and in real time through the environmental model, and timely grasp the status of various equipment and facilities in the target scene, thereby helping to improve the effectiveness of the equipment control service. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions provided by this application, the following briefly introduces the drawings required for use in describing each embodiment of this application.

[0027] Figure 1 is a schematic diagram of an implementation environment involved in an embodiment of the present application;

[0028] Figure 2 is a flow chart showing a method for generating an environment model according to an exemplary embodiment;

[0029] Figure 3 is a schematic diagram of a space layout diagram corresponding to a home scene according to an exemplary embodiment;

[0030] Figure 4a is a schematic diagram of an environment model of a home scene according to an exemplary embodiment;

[0031] Figure 4b is a schematic diagram of an environment model of a home scene according to another exemplary embodiment;

[0032] Figure 5 yes Figure 2 A flowchart of an embodiment corresponding to step 240 in an embodiment;

[0033] Figure 6 yes Figure 2 A flowchart of an embodiment corresponding to step 220 in an embodiment;

[0034] Figure 7 yes Figure 2 A flowchart of another embodiment corresponding to step 220 in the embodiment;

[0035] Figure 8 yes Figure 2 A flowchart of another embodiment corresponding to step 240 in the embodiment;

[0036] Figure 9 yes Figure 2 A flowchart of a process diagram of generating a spatial layout diagram corresponding to a target scene before step 200 in a corresponding embodiment;

[0037] Figure 10 is a schematic diagram illustrating a process of generating a spatial layout diagram corresponding to a target scene according to an exemplary embodiment;

[0038] Figure 11 is a flow chart showing a method for displaying an environment model according to an exemplary embodiment;

[0039] Figure 12 is a structural block diagram of a device for generating an environment model according to an exemplary embodiment;

[0040] Figure 13 is a structural block diagram of an environment model display device according to an exemplary embodiment;

[0041] Figure 14 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0042] Figure 15 The figure is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0044] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0045] As mentioned above, the relevant technology still lacks a solution for intuitively monitoring each smart device in the scene, which in turn affects the effectiveness of the device control service.

[0046] Taking the smart home scene as an example, most technical solutions only obtain the floor plan consisting of the edges of the home rooms, and are unable to obtain the locations of various smart devices in the smart home scene in the smart home scene, and thus cannot associate various smart devices with the home room where the smart home scene is located, and thus cannot realize intuitive monitoring based on the home room for various smart devices in the smart home scene.

[0047] Currently, the monitoring of various smart devices in smart home scenarios mainly relies on user terminals. For example, in a smart home scenario, the user terminal determines whether the air conditioner in the living room is turned on, as well as the temperature and operating mode of the air conditioner based on the device status data reported by the air conditioner, and displays it. However, under this monitoring solution, users cannot intuitively monitor the various smart devices in the smart home scenario, which affects the effectiveness of the device control service. In addition, it is difficult to effectively monitor non-smart devices in smart home scenarios, because non-smart devices are also important environmental factors. This further increases the difficulty of improving the effectiveness of device control services for the entire house.

[0048] As can be seen from the above, the related art still has the defect of low effectiveness of device control services.

[0049] To this end, the environmental model generation method provided in the present application can generate an environmental model corresponding to the target scene to describe the state of the second target object in the target scene. Through the environmental model, each second target object in the target scene is associated with the spatial layout diagram corresponding to the target scene, thereby realizing a device monitoring solution based on spatial positioning, thereby effectively improving the effectiveness of the device control service. Accordingly, the environmental model generation method is suitable for an environmental model generation device, and the environmental model generation device can be deployed on an electronic device, for example, the electronic device can be a smart phone, a tablet computer, etc.

[0050] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0051] Figure 1 The schematic diagram of the implementation environment involved in the generation method of an environment model. The implementation environment at least includes a user terminal 110, a smart device 130, a server 170, and a network device. Figure 1 In the embodiment, the network devices include a gateway 150 and a router 190, which is not specifically limited here.

[0052] Among them, the user terminal 110, which can also be considered as a user end or terminal, can deploy (also understood as install) the client associated with the smart device 130. This user terminal 110 can be an electronic device such as a smart phone, tablet computer, laptop computer, desktop computer, smart control panel, other devices with display and control functions, etc., which are not limited here.

[0053] Among them, the client is associated with the smart device 130. In essence, the user registers an account in the client and configures the smart device 130 in the client. For example, the configuration includes adding a device identifier to the smart device 130, so that when the client is run in the user terminal 110, it can provide the user with device display, device control and other functions about the smart device 130. This client can be in the form of an application or a web page. Accordingly, the interface for device display on the client can be in the form of a program window or a web page, which is not limited here.

[0054] The smart device 130 is deployed in the gateway 150, and communicates with the gateway 150 through its own configured communication module, and is thereby controlled by the gateway 150. It should be understood that the smart device 130 generally refers to one of a plurality of smart devices 130, and the embodiment of the present application only takes the smart device 130 as an example, that is, the embodiment of the present application does not limit the number and device types of smart devices deployed in the gateway 150. In an application scenario, the smart device 130 accesses the gateway 150 through a local area network, and is thereby deployed in the gateway 150. The process of the smart device 130 accessing the gateway 150 through the local area network includes: the gateway 150 first establishes a local area network, and the smart device 130 joins the local area network established by the gateway 150 by connecting to the gateway 150. This local area network includes but is not limited to: ZIGBEE or Bluetooth. The smart device 130 may be a smart printer, a smart fax machine, a smart camera, a smart air conditioner, a smart door lock, a smart light, or a human body sensor equipped with a communication module, a door and window sensor, a temperature and humidity sensor, a water sensor, a natural gas alarm, a smoke alarm, a wall switch, a wall socket, a wireless switch, a wireless wall switch, a magic cube controller, a curtain motor, a millimeter-wave radar, or the like. It should be noted that the millimeter-wave radar can transmit FMCW signals to other smart devices 130 and then receive echo signals reflected by other smart devices 130, or capture micro-motion signals generated by other smart devices 130 during operation and receive echo signals. Because the echo signals cover all distance ranges, any moving object within the millimeter-wave radar's detection range can be detected. This characteristic of millimeter-wave radar enables effective monitoring of other smart devices. The micro-motion signals refer to vibration waves generated by the operation of other smart devices 130. For example, micro-motion signals can be generated by smart air conditioners, fans, vacuum cleaners, air purifiers, and even computer cases with rotating fans.

[0055] The interaction between the user terminal 110 and the smart device 130 can be achieved through a local area network or a wide area network. In one application scenario, the user terminal 110 establishes a communication connection between the router 190 and the gateway 150 by means of a wired or wireless method. For example, the wired or wireless method includes but is not limited to WIFI, so that the user terminal 110 and the gateway 150 are deployed in the same local area network, thereby enabling the user terminal 110 to interact with the smart device 130 through the local area network path. In another application scenario, the user terminal 110 establishes a communication connection between the server 170 and the gateway 150 by means of a wired or wireless method. For example, the wired or wireless method includes but is not limited to 2G, 3G, 4G, 5G, WIFI, etc., so that the user terminal 110 and the gateway 150 are deployed in the same wide area network, thereby enabling the user terminal 110 to interact with the smart device 130 through the wide area network path.

[0056] The server side 170 can also be considered as a cloud, cloud platform, platform side, service side, etc. The server side 170 can be a single server, a server cluster composed of multiple servers, or a cloud computing center composed of multiple servers, so as to better provide background services to a large number of user terminals 110. For example, the background services include device control services.

[0057] In an application scenario, after the user terminal 110 obtains the spatial layout diagram corresponding to the target scene based on the relationship between the movement trajectory of the first target object in the target scene and the various spaces in the target scene, it can display the spatial layout diagram corresponding to the target scene, and then identify the second target objects deployed in the target scene through the echo signal received by the smart device 130, and obtain the position of each second target object in the target scene, and then generate an environmental model of the target scene based on the position of each second target object in the target scene and the spatial layout diagram corresponding to the target scene. The environmental model is used to describe the status of each second target object in the target scene, and each second target object in the target scene is monitored more effectively and intuitively, thereby providing users with more effective device control services.

[0058] Of course, in other application scenarios, the above-mentioned process of generating the environmental model completed by the user terminal 110 can also be implemented by the server side 170. At this time, after the server side 170 generates the environmental model of the target scene, the environmental model of the target scene is sent to the user terminal 110 through the wide area network path, and the environmental model is displayed on the user terminal 110, so that the user terminal 110 can provide the user with more effective device control services based on the environmental model.

[0059] See also Figure 2 The embodiment of the present application provides a method for generating an environment model, which is applicable to an electronic device, which may be Figure 1 A user terminal 110 of the illustrated implementation environment.

[0060] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0061] like Figure 2 As shown, the method may include the following steps:

[0062] Step 200: Display a spatial layout diagram corresponding to the target scene.

[0063] The target scene can be an indoor or outdoor scene, such as a home, office, or park. The spatial layout diagram describes the location, size, and other spatial information of each space in the target scene. For example, a space could be a living room or bedroom in a home, a conference room in an office, or an entertainment area in a park. In other words, the spatial layout diagram reflects the boundary locations and orientation relationships of each space in the target scene.

[0064] For example, the space layout diagram can be a floor plan. Figure 3 , a spatial layout diagram corresponding to the target scene is shown, wherein the target scene is a smart home scene, and each space in the smart home scene includes at least a second bedroom, a kitchen, a toilet, a washroom, a living room and a dining room, two master bedrooms, and a balcony, etc. It can be seen that the boundary position and orientation relationship of each space in the smart home scene can be reflected through this spatial layout diagram.

[0065] Regarding the generation of the spatial layout diagram, in one possible implementation, the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and the various spaces in the target scene. The first target object refers to an object that can move within the target scene, such as a person or a robot. The trajectory can refer to the movement trajectory of the first target object within a certain area of the target scene or the movement trajectory of the first target object within a certain location in the target scene, without limitation herein.

[0066] In one possible implementation, infrared technology can be used to determine the spatial layout of the target scene. For example, the spatial layout can be determined through the following process: use near-infrared rays to scan the room to obtain a preliminary floor plan of the room; identify jumps in the outline of the floor plan, remove the jump outline, and mark the door position and door width at the jump position; obtain the floor plan inside the room wall; when a room has multiple doorways, number the multiple doorways; measure the wall thickness: use a distant external line to measure the thickness of the wall where the doorway is located; room splicing: use the doorway as the splicing reference, select the doorways corresponding to the two rooms as the splicing basis for splicing, and use the wall thickness as the interval distance to splice the floor plans of adjacent rooms to obtain a complete floor plan.

[0067] In one possible implementation, millimeter-wave radar can also be used to determine the spatial layout of the target scene. For example, this can be determined through the following process: filtering static points in the point cloud data to retain dynamic points; binning the dynamic point data to obtain a dynamic point cloud; tracking and monitoring the dynamic point cloud to obtain the tester's movement trajectory within the test area; and analyzing and processing the movement trajectory to calculate the left, right, and front boundaries of the test area. By continuously observing the person's back-and-forth movement trajectory, the left, right, and front boundaries of the house each time the person passes can be calculated. Through multiple sets of measurements, the front, left, and right boundaries of the test area can be ultimately determined.

[0068] For electronic devices, after obtaining the spatial layout diagram corresponding to the target scene, the spatial layout diagram can be displayed on the display screen configured by the electronic device, so as to subsequently generate an environmental model of the target scene in combination with the position of the second target object in the target scene.

[0069] Step 220 : Identify each second target object deployed in the target scene, and obtain the position of each identified second target object in the target scene.

[0070] Among them, the second target object refers to the equipment or facilities deployed in the target scene. In one possible implementation method, the second target object may refer to a device that can generate a micro-motion signal when working, that is, an electrically controlled object. For example, the electrically controlled object may be a smart door lock, smart (electric) curtains, desktop computers, range hoods, electric fans, air conditioners, etc.; the second target object may also be a facility that cannot generate a micro-motion signal, that is, a non-electrically controlled object. For example, the non-electrically controlled object may be a sofa, wardrobe, bay window, etc. in the target scene, which is not limited here.

[0071] The identification of the second target object may include identifying the second target object's location in the target scene, object type, and operating status, etc., which are not limited here. The object type is used to distinguish different second target objects. Different second target objects have different object types, such as air conditioner a and plant c. If the second target objects are the same, they have the same object type, such as air conditioner a and air conditioner b. The operating status is used to describe the operating status of the second target object. For example, the operating status includes on, off, faulty, etc.

[0072] If the second target object is an electrically controlled object, in one possible implementation, a radar device is used to identify the position of the electrically controlled object in the target scene. In another possible implementation, a radar device is used to identify the working state of the electrically controlled object.

[0073] If the second target object is a non-electrically controlled object, in one possible implementation, the position of the first target object interacting with the non-electrically controlled object is identified by a radar device, thereby indirectly identifying the position of the non-electrically controlled object in the target scene.

[0074] If the second target object is a smart device, in one possible implementation, the device type of the second target object is identified by combining the device status data reported by the smart device and the micro-motion signal generated when the second target object is working and captured by the radar device.

[0075] In a possible implementation, the object type of the second target object is identified through user operation (manual marking), image recognition, or the like.

[0076] Step 240 : Generate an environment model of the target scene according to the position of each second target object in the target scene and the spatial layout diagram corresponding to the target scene.

[0077] The environment model is used to describe the state of the first target object and / or the second target object in the target scene. This state may refer to the object type of the first target object and / or the second target object in the target scene, the operating state of the first target object and / or the second target object in the target scene, or the position, trajectory, motion state, etc. of the first target object and / or the second target object in the target scene, without specific limitation herein.

[0078] In one possible implementation, Figure 4a A schematic diagram showing the environment model of a smart home scene. Figure 4a It can be seen that the environmental model of the smart home scene can reflect the position, object type and working status of each second target object in the smart home scene. For example, the second target object 301 and the second target object 302 have the same object type, both indicating that the second target object is an air conditioner, the position of the air conditioner 301 is in the lower left corner of the living room and dining room, and the working status of the air conditioner 301 is off; the position of the air conditioner 302 is in the lower left corner of the master bedroom, and the working status of the air conditioner 302 is on; the object type of the second target object 303 indicates that the second target object is a green plant, the position of the green plant 303 is in the upper left corner of the living room and dining room, and the green plant 303 is not in a working state.

[0079] In one possible implementation, Figure 4b A schematic diagram showing the environment model of a smart home scenario, Figure 4b The diagram part 401 of the environmental model of the smart home scene is shown, which includes the position, object type and working status of each second target object in the smart home scene, and may also include the trajectory and / or motion status of each first target object in the smart home scene. Figure 4bIt also includes a status display area 402 for each first target object and / or each second target object, in which the status of each first target object in the smart home scene and the status of each second target object in the smart home scene can be displayed separately.

[0080] For example, in the status display area 402, the status of the user as the first target object can be displayed; specifically, it can include the user's trajectory in the smart home scene, and motion status, such as still, sitting, falling, etc., which can be specifically obtained by detecting the human body status in the smart home scene. In the status display area 402, the status of various detectable devices and / or facilities as second target objects in each area space can also be displayed. For example, it can be detected that all doors are closed, the master bedroom window is open, the master bedroom fan is on, the range hood is on, the computer is off, the master bedroom air conditioner is on, the living room purifier is on, the master bedroom purifier is off, the living room air conditioner is off, the bathroom exhaust fan is off, etc. Therefore, by displaying the status of each target object in each area of the target scene in the form of icons and text in another area, the status of each first target object and the second target object in the target scene can be more intuitively known.

[0081] In a possible implementation, the environment model is implemented by displaying object labels of each second target object in the spatial layout diagram. Figure 5 As shown, step 240 may include the following steps:

[0082] Step 500: Determine the corresponding area of each second target object in the spatial layout diagram according to the position of each second target object in the target scene.

[0083] That is, the position of the second target object in the target scene is matched to the coordinate system of the spatial layout diagram, so as to map the position of the second target object in the target scene to the corresponding area in the spatial layout diagram.

[0084] Step 520 : Display the object mark of each second target object in the corresponding area of the spatial layout diagram to generate an environment model of the target scene.

[0085] The object tag of the second target object is used to indicate the state of the second target object.

[0086] In a possible implementation, the object mark of the second target object may be an icon of different shapes and colors, and the shape and color of the object mark respectively indicate the object type and working status of the second target object. Figure 4aFor example, object marker 301 has the shape of an air conditioner and is light gray, indicating that the second target object is an air conditioner and its operating state is off, i.e., "AC Off"; object marker 302 has the shape of an air conditioner and is dark black, indicating that the second target object is an air conditioner and its operating state is on, i.e., "AC On"; object marker 303 has the shape of a plant, indicating that the second target object is a plant. It should be noted that some second target objects do not have an operating state. Furthermore, the position of an object marker in the environment model can map to the position of the corresponding second target object in the target scene. Thus, the environment model can also reflect the position of the second target object in the target scene.

[0087] Through the above process, a device monitoring solution based on spatial positioning is provided, that is, the environmental model is used to associate each second target object in the target scene with the spatial layout diagram corresponding to the target scene, so that the second target objects (that is, various equipment and facilities) in the target scene can be monitored intuitively, dynamically and in real time through the environmental model, and the working status of various equipment and facilities in the target scene can be grasped in time, which is conducive to improving the effectiveness of equipment control services.

[0088] See also Figure 5 In an exemplary embodiment, after step 520, the method may further include the following steps:

[0089] Step 540 : Identify a second target object newly deployed in the target scene based on the association relationship between each displayed second target object and the signal feature of the echo signal.

[0090] The echo signal is obtained by the radar device capturing the micro-motion signal generated when the second target object is working.

[0091] It will be appreciated that if the second target objects in the target scene are of different object types, the echo signals generated by the micro-motion signals generated by the second target objects during operation will also be different. Based on this, second target objects of the same object type can be identified based on the second target objects already displayed in the environment model. Specifically, for each second target object already displayed in the environment model, the signal characteristics of the echo signals generated have already been associated with the corresponding second target object, and based on this association, second target objects newly deployed in the target scene can be identified.

[0092] For example, the object types of range hoods and exhaust fans are different, and the echo signals formed by the micro-motion signals generated by their operation are also different. Therefore, they are associated with the signal characteristics of the corresponding echo signals. When a new exhaust fan is deployed in the target scene, the object type of the newly deployed exhaust fan can be identified through the echo signal of the newly deployed exhaust fan based on this association relationship.

[0093] Step 560: Display the object mark of the newly deployed second target object in the corresponding area of the spatial layout diagram, and update the environment model of the target scene.

[0094] In one possible implementation, the position of the newly deployed second target object in the target scene is obtained based on the echo signal generated by the newly deployed second target object to determine the corresponding area of the newly deployed second target object in the spatial layout diagram, and then the environmental model of the target scene is updated, that is, the newly deployed second target object is marked in the corresponding area in the spatial layout diagram.

[0095] Of course, in addition to updating the environmental model of the target scene by identifying the newly deployed second target object, the environmental model can also be updated by identifying the working status of the second target object that has been displayed in the environmental model. For example, each time the exhaust fan is turned on, the millimeter-wave radar receives the same echo signal, and the echo signal disappears after it is turned off. An association is established between the echo signal and the working status of the exhaust fan. The exhaust fan can be identified as currently being in the on or off working state based on whether there is a similar echo signal. Based on this, the environmental model is updated to ensure that the environmental model dynamically reflects the real-time working status of the second target object in the target scene.

[0096] Through the above process, the signal characteristics of the echo signal reflected / scattered by the displayed second target object are learned, and the signal characteristics of the specific echo signal are associated with the object type and working status of the second target object. Therefore, the object type and / or working status of the newly deployed second target object can be identified through the signal characteristics of the echo signal reflected / scattered by the newly deployed second target object, and the environmental model is updated, thereby fully ensuring that the environmental model can dynamically reflect the changes and real-time working status of the second target object in the target scene, further enhancing the real-time monitoring capability of various equipment and facilities in the target scene, and helping to further improve the effectiveness of equipment control services.

[0097] As mentioned above, the second target object refers to the equipment or facilities deployed in the target scene. Specifically, the second target object includes an electrically controlled object and a non-electrically controlled object.

[0098] Electronically controlled objects are devices that generate micromotion signals when in operation, while non-electronically controlled objects are devices that do not. Micromotion signals are vibration waves generated actively or passively by the second target object. Examples include plants swaying in the wind, drawn curtains, air conditioners, fans, sweepers, pots vibrating due to heating, air purifiers, and computer cases with rotating fans.

[0099] The following combination Figures 6 to 8, the identification of each second target object deployed in the target scene is described in detail:

[0100] like Figure 6 As shown, in an exemplary embodiment, when the second target object is an electrically controlled object, step 220 may include the following steps:

[0101] Step 600 : Positioning the electrically controlled object based on the echo signal, identifying the position of the electrically controlled object in the target scene, and determining a corresponding area of the electrically controlled object in the spatial layout diagram.

[0102] Among them, the echo signal is obtained by the radar equipment capturing the micro-motion signal generated when the electronically controlled object is working.

[0103] In this way, radar is used to identify and locate the position of the electronically controlled object in the target scene, and a precise matching relationship between the electronically controlled object in the target scene and the spatial layout diagram is established, thereby making the environmental model more accurate and enhancing environmental supervision capabilities.

[0104] exist Figure 6 In the embodiment, when the second target object is an electrically controlled object, step 220 may further include the following steps:

[0105] Step 620 : Based on the radar device capturing the micro-motion signal generated when the electrically controlled object is working, it is identified that the working state of the electrically controlled object is the on state, so as to mark the electrically controlled object in the on state in the corresponding area determined by the spatial layout diagram.

[0106] That is to say, when the electronically controlled object starts to work, a specific micro-motion signal will always appear in the corresponding area. Then, based on the radar equipment's capture of the micro-motion signal generated when the electronically controlled object is working, it can be identified that the working state of the electronically controlled object is on.

[0107] In this way, the working status of the electronically controlled object is identified, which serves as the basis for generating the environmental model, making it easier to grasp the working status of each second target object in the target scene in real time and dynamically, thereby effectively monitoring the second target object and enhancing the monitoring capability of equipment and facilities.

[0108] When the second target object is a non-electrically controlled object, such as Figure 7 As shown, step 220 may further include the following steps:

[0109] Step 700 : Positioning a first target object interacting with a non-electrically controlled object based on an echo signal to obtain a position of the first target object in a target scene.

[0110] The echo signal is obtained when the transmission signal of the radar device is reflected by the first target object.

[0111] The first target object refers to an object that can move in the target scene. For example, the first target object can be a person, a robot, etc.

[0112] A non-electrically controlled object refers to a facility that cannot generate a micro-motion signal, for example, a non-electrically controlled object may be a sofa.

[0113] Step 720 : Identify the position of the non-electrically controlled object in the target scene according to the position of the first target object in the target scene, so as to determine the corresponding area of the non-electrically controlled object in the spatial layout diagram.

[0114] For example, as a person interacts with a sofa, such as sitting on the sofa, the person is located based on the echo signal to obtain the person's position in the target scene. The edge position of the sofa area is further calculated based on the person's position in the target scene and the person's approximate height, thereby identifying the position of the sofa in the target scene.

[0115] During the above process, the inventors realized that non-electrically controlled objects cannot generate micro-motion signals, and the radar equipment has poor recognition and positioning capabilities for stationary objects, that is, it is difficult to directly locate non-electrically controlled objects. Therefore, for non-electrically controlled objects, the first target object that interacts with the non-electrically controlled object is located to obtain the position of the first target object, and indirectly identify the position of the non-electrically controlled object in the target scene, thereby overcoming the defect of difficulty in effectively monitoring non-electrically controlled objects and fully ensuring that non-electrically controlled objects, as important environmental influencing factors, can also effectively provide device control services for users.

[0116] There is another case where the second target object is an unknown smart device. Figure 8 As shown, in an exemplary embodiment, step 220 may further include the following steps:

[0117] In step 800 , if it is detected that the working state of the smart device changes from the off state to the on state, the area range of the smart device in the target scene is determined.

[0118] It should be noted that the smart device capable of working status detection here is a known smart device in the target scene, that is, the smart device has been deployed in Figure 1 In the gateway 150 of the illustrated implementation environment, the smart device can report device status data via the gateway 150 so that the server 170 / user terminal 110 can detect whether the working status of the smart device has changed.

[0119] For example, the working state of the smart air conditioner in the bedroom changes from the off state to the on state. The smart air conditioner reports its own device status data through the network device. The device status data is used to indicate the working state of the smart air conditioner. After detecting the change in the working state, the server or user terminal first determines the area range of the smart air conditioner in the smart home scene, so as to subsequently determine whether there is an unknown smart device in the area range of the bedroom, that is, the second target object that generates a micro-motion signal when working, and then determine the object type of the second target object.

[0120] Step 820: Within the determined area, if the radar device captures the micro-motion signal generated when the second target object is working, the device type of the smart device is identified as the object type of the second target object, so as to mark the second target object according to the device type of the smart device in the spatial layout diagram.

[0121] Still using the above example to illustrate, within the area of the bedroom, the radar device determines that it has captured the micro-motion signal generated by the operation of the second target object through the received echo signal, and based on the change in the working state of the corresponding smart device detected in step 800, the smart device with the change in working state is associated with the second target object detected by the radar device, that is, it is determined that the second target object is the smart air conditioner in the bedroom, thereby completing the identification of the object type of the second target object.

[0122] In the above process, based on the properties of the Internet of Things for smart devices, the device status data of the smart devices deployed in the gateway can be obtained, and the unknown smart devices and known smart devices can be associated with each other. The object type of the second target object, that is, the device type of the corresponding smart device, can be determined, which is further conducive to the effective monitoring of each smart device in the target scene, and thus helps to improve the effectiveness of the device control service.

[0123] For the second target object in the target scene, in addition to automatic recognition by the electronic device, it can also be identified by other means. In an exemplary embodiment, step 220 can also include the following steps:

[0124] In response to a type tagging operation on the second target object, the object type of the second target object is identified; or in response to a recognition event triggered by the image acquisition device on the second target object, the object type of the second target object is identified.

[0125] In one possible implementation, the type tagging operation for the second target object may be a manual tagging operation by a user, thereby identifying the object type of the second target object through manual tagging by the user. The manual tagging operation by the user may be triggered actively by the user or passively, for example, by generating a tag prompt message to prompt the user to manually tag the type of the second target object, although this is not limited herein.

[0126] The image acquisition device can be a camera, a smartphone equipped with a camera, etc. The recognition event triggered by the image acquisition device for the second target object can refer to the image acquisition device acquiring and photographing the image to be identified containing the second target object, and performing image recognition on the image to be identified containing the second target object, thereby identifying the object type of the second target object.

[0127] By identifying the object type of the second target object through a type marking operation on the second target object or an identification event triggered by an image acquisition device on the second target object, different identification methods are further provided to enhance the environmental model's ability to monitor the target scene equipment and facilities (i.e., the second target object).

[0128] In conjunction with the above-mentioned embodiments, full consideration is given to the possible existence of second target objects of different forms in the target scene, and different identification methods are provided for different second target objects to meet the identification requirements for different second target objects, thereby being able to identify more equipment and facilities in the target scene, and more accurately identify each second target object, obtain the position, working status and object type of each second target object, and thus generate an environmental model, which is convenient for users to intuitively understand the position, object type or working status of each second target object in the target scene, and can dynamically and real-time monitor the equipment and facilities in the target scene.

[0129] See also Figure 9 In an exemplary embodiment, before step 200, the method may further include the following steps:

[0130] Step 900: Obtain the trajectory information of the first target object in each space in the target scene, draw and display the trajectory diagram of each space.

[0131] The trajectory information is used to indicate the movement trajectory of the first target object in each space.

[0132] In one possible implementation, a radar device is used to locate a first target object to obtain trajectory information of the first target object. Specifically, in step 900, obtaining trajectory information of the first target object in each space within the target scene includes tracking the trajectory of the first target object based on an echo signal to obtain the trajectory information of the first target object in each space. The echo signal is obtained by reflecting a transmitted signal of the radar device off the first target object.

[0133] like Figure 10 As shown in FIG, a schematic diagram of the process of generating a corresponding spatial layout diagram from a target scene in an example is shown, where P1 is the target scene, P2 is a trajectory diagram with entrance and exit locations marked, and P3 is a spatial layout diagram. As can be seen from P2, the trajectory diagram can be a boundary outline diagram of each space in the target scene.

[0134] Step 920: Mark the entrance and exit locations of each space on the trajectory map of each space.

[0135] In a possible implementation, the entrance and exit locations of each space are determined by analyzing the opening and closing actions of the first target object in each space. Specifically, step 920 may include:

[0136] Step 921 : In response to a space opening and closing event triggered in each space, respectively determine the position of the first target object in each space when the space opening and closing event is triggered.

[0137] The space opening and closing event refers to a door opening and closing event. The first target object may be a person. When the space opening and closing event is triggered, it refers to the moment when the person opens / closes the door.

[0138] Correspondingly, when a person opens / closes the door of a space, the person's position can be located by radar equipment and used as the entrance and exit position of the space.

[0139] Step 922: Mark the entrance and exit positions of each space in the corresponding area of the trajectory map of each space according to the determined position.

[0140] It can be understood that when a person opens / closes a door, the person's position coincides with the entrance and exit positions of each space. Therefore, the entrance and exit positions of each space can be marked separately in the trajectory diagram of each space according to the person's position at this time.

[0141] like Figure 10 As shown, because the entrance and exit positions connect adjacent spaces, one entrance and exit position is marked on the trajectory diagrams of two different spaces. In a possible implementation, the entrance and exit positions are determined by manually marking the connecting positions. Specifically, step 920 may also include:

[0142] Step 924 : In response to the marking operation on the connected location, determine two adjacent spaces associated with the connected location.

[0143] The connected position is used to indicate the position of the first target object when it enters or exits between two adjacent spaces.

[0144] The marking operation for the connection location may be performed manually by the user.

[0145] Step 925: On the trajectory diagram of the two adjacent spaces, mark the connecting positions as the entrance and exit positions of the two adjacent spaces.

[0146] Step 940: obtain a spatial layout diagram by matching the trajectory diagrams of each space with marked entrance and exit locations.

[0147] Among them, matching is to connect the trajectory diagrams of each space with marked entrance and exit positions according to the spatial orientation relationship, entrance and exit positions, etc. to obtain a spatial layout diagram.

[0148] See also Figure 10 , which shows a schematic diagram of generating a spatial layout diagram of a target scene in an exemplary embodiment.

[0149] The target scene P1 is a smart home scene. The first target object (i.e., user) is allowed to move in each space of the target scene P1. The trajectory information of the first target object (user) in each space is obtained based on radar tracking. For example, Figure 10 In P1, the trajectory information includes the user's movement trajectory in each space of the target scene (the figure shows the user's movement route map). A trajectory map P2 for each space is then drawn and displayed. By locating the first target object when a space opening or closing event is triggered, the entrance and exit locations of each space are marked in the corresponding area of the trajectory map. Finally, based on the entrance and exit locations and the user's movement direction, the trajectory maps of each space with marked entrance and exit locations are matched to obtain a spatial layout map.

[0150] Under the influence of the above-mentioned embodiments, a spatial layout diagram is generated based on the relationship between the movement trajectory of the first target object in the target scene and the various spaces in the target scene, providing a method for generating a spatial layout diagram by tracking the target object. Compared with the solution of generating a spatial layout diagram in the related technology, it is simpler, more convenient and more efficient.

[0151] See also Figure 11 The embodiment of the present application provides a method for displaying an environment model, which is applicable to an electronic device, which may be Figure 1 A user terminal 110 of the illustrated implementation environment.

[0152] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0153] like Figure 11 As shown, the method may include the following steps:

[0154] Step 1100: Display an environment model corresponding to the target scene.

[0155] First, it should be noted that the environment model includes the state of each first target object and / or each second target object in the target scene. This state can refer to the object type of the first target object and / or the second target object in the target scene, the operating state of the first target object and / or the second target object in the target scene, or the position, trajectory, motion state, etc. of the first target object and / or the second target object in the target scene, without specific limitation herein.

[0156] The generation of the environment model of the target scene is based on the spatial layout diagram of the target scene and the positions of the second target objects in the target scene.

[0157] Secondly, the first target object refers to an object that can move within the target scene. For example, the first target object can be a person, a robot, etc., so that a spatial layout diagram can be generated based on the relationship between the trajectory of the first target object in the target scene and the various spaces in the target scene, which in turn serves as the basis for generating an environmental model of the target scene. The trajectory can refer to the movement trajectory of the first target object in a certain area of the target scene, or it can refer to the movement trajectory of the first target object in a certain position in the target scene, which is not limited here.

[0158] The second target object refers to the equipment or facility deployed in the target scene. In one possible implementation, the second target object may refer to a device that can generate a micro-motion signal when working, that is, an electrically controlled object. For example, the electrically controlled object may be a smart door lock, smart (electric) curtains, desktop computers, range hoods, electric fans, air conditioners, etc.; the second target object may also be a facility that cannot generate a micro-motion signal, that is, a non-electrically controlled object. For example, the non-electrically controlled object may be a sofa, wardrobe, bay window, etc. in the target scene, which is not limited here.

[0159] Step 1120 : If it is recognized that the status of each first target object and / or each second target object has changed, the updated status of the first target object and / or the second target object is displayed in the environment model.

[0160] Among them, the status of the first target object and / or the second target object can be represented by different object marks, which can be icons of different shapes and colors, and the status of the first target object and / or the second target object can be represented by icons of different shapes and colors, and the status includes but is not limited to object type, working status, location, etc.

[0161] Please refer back Figure 4a Object marker 301 has the shape of an air conditioner and is light gray, indicating that the second target object is an air conditioner and its operating state is off, i.e., "AC Off"; object marker 302 has the shape of an air conditioner and is dark black, indicating that the second target object is an air conditioner and its operating state is on, i.e., "AC On"; object marker 303 has the shape of a plant, indicating that the second target object is a plant. It should be noted that some second target objects do not have an operating state. Furthermore, the position of an object marker in the environment model can map to the position of the corresponding second target object in the target scene. Thus, the environment model essentially reflects the position of the second target object in the target scene.

[0162] Please refer back Figure 4b A schematic diagram showing the environment model of a smart home scenario, Figure 4b The diagram part 401 of the environmental model of the smart home scene is shown, which includes the position, object type and working status of each second target object in the smart home scene, and may also include the trajectory and / or motion status of each first target object in the smart home scene. Figure 4b The smart home scene also includes a status display area 402 for each first target object and / or each second target object. This area can separately display the status of each first target object in the smart home scene, as well as the status of each second target object in the smart home scene. By displaying the status of each target object in each area of the target scene in the form of icons and text in separate areas, the status of each first target object and second target object in the target scene can be more intuitively understood.

[0163] Through the above embodiments, the environmental model is displayed on the interface of the electronic device, and based on the status changes of the first target object and / or each second target object, the displayed environmental model is updated in a timely manner, so that the user can more intuitively control the status of each first / second target object in the target scene in real time.

[0164] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0165] 1. The environmental model is used to describe the object type and / or status of the second target object in the target scene, and can realize equipment and facility supervision based on spatial positioning, so that users can remotely and intuitively grasp the real-time status of the equipment and facilities in the target scene, so that users can further manage these equipment and facilities.

[0166] 2. Generate a spatial layout diagram of the target scene based on the relationship between the movement trajectory of the first target object in the target scene and each space in the target scene, identify each second target object deployed in the target scene to obtain the position of the second target object in the target scene, and then generate an environmental model of the target scene based on the spatial layout diagram of the target scene and the position of each second target object in the target scene. The environmental model is used to describe the object type and / or state of the second target object in the target scene. Thus, the second target object can be associated with the spatial layout diagram, providing an environmental supervision method based on spatial positioning, which can intuitively, dynamically and in real time supervise various equipment and facilities in the target scene through the environmental model, obtain the status of each second target object (i.e., each equipment and facility) in the target scene in real time, and improve the real-time monitoring and management capabilities of the equipment and facilities in the target scene. This application solution can help users intuitively supervise the equipment and facilities in the target scene and understand the real-time status of different equipment, so as to facilitate management of the equipment in the target scene.

[0167] 3. Learn the signal characteristics of the echo signal reflected / scattered by the displayed second target object, and establish an association between the signal characteristics of the specific echo signal and the object type and status of the second target object, so as to identify the object type and / or status of the newly deployed second target object through the signal characteristics of the echo signal reflected / scattered by the newly deployed second target object, and update the environmental model accordingly; of course, for the second target object that has been displayed in the environmental model, its status can be identified through the signal characteristics of its echo signal, and the environmental model can be updated. Updating the environmental model ensures that the environmental model dynamically reflects the real-time status of the second target object in the target scene, and can also promptly present the newly deployed second target object in the environmental model after the new second target object is deployed in the target scene, further enhancing the real-time supervision capability of the target scene.

[0168] 4. Considering that non-electronically controlled objects cannot generate micro-motion signals, and the radar's ability to identify and locate stationary objects is poor, it is difficult to directly locate non-electronically controlled objects. Therefore, for non-electronically controlled objects, the position of the first target object interacting with the non-electronically controlled object is located to obtain the position of the first target object, and the position of the non-electronically controlled object in the target scene is indirectly identified, overcoming the defect of difficulty in locating non-electronically controlled objects.

[0169] 5. Based on the IoT attributes of smart devices, their data and information can be accessed. Upon detecting a change in the smart device's state from off to on, the radar captures the micro-motion signal generated by the second target object operating within the corresponding area, indicating the second target object has been discovered. By associating the discovered second target object with the smart device, the object type of the second target object, i.e., the device type of the corresponding smart device, can be determined. The above embodiment provides a smart device identification method that meets the identification requirements for different second target objects.

[0170] 6. Taking into full consideration the possible existence of second target objects of different forms in the target scene, different identification methods are provided for different second target objects to meet the identification needs of different second target objects, so that more equipment and facilities in the target scene can be identified. It can also more accurately identify each second target object, obtain the position, status and object type of each second target object, and thus generate an environmental model, which is convenient for users to intuitively understand the position, object type or status of each second target object in the target scene, and can dynamically and real-time monitor the equipment and facilities in the target scene.

[0171] 7. A spatial layout diagram is generated based on the movement trajectory of the first target object in the target scene and the relationship between the various spaces in the target scene, providing a method for generating a spatial layout diagram by tracking the target object. Compared with the solution of generating a spatial layout diagram in the related art, it is simpler, more convenient and more efficient.

[0172] The following is an embodiment of the device of the present application, which can be used to perform the method for generating an environment model involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method for generating an environment model involved in the present application.

[0173] See also Figure 12 In an embodiment of the present application, a device 1200 for generating an environment model is provided, including but not limited to a layout diagram display module 1210 , an object recognition module 1230 , and a model generation module 1250 .

[0174] The layout diagram display module 1210 is used to display a spatial layout diagram corresponding to the target scene; the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and each space in the target scene.

[0175] The object recognition module 1230 is configured to recognize each second target object deployed in the target scene and obtain a position of each recognized second target object in the target scene.

[0176] The model generation module 1250 is used to generate an environmental model of the target scene based on the position of each second target object in the target scene and the spatial layout diagram corresponding to the target scene; the environmental model is used to describe the state of the first target object and / or the second target object in the target scene.

[0177] In an exemplary embodiment, the model generation module 1250 may include: an area determination unit 1251, used to determine the corresponding area of each second target object in the spatial layout diagram based on the position of each second target object in the target scene; a mark display unit 1252, used to display the object mark of each second target object in the corresponding area in the spatial layout diagram to generate an environmental model of the target scene; the object mark of the second target object is used to indicate the status of the second target object.

[0178] In an exemplary embodiment, the device 1200 also includes: an object re-identification module 1270, which is used to identify the second target object newly deployed in the target scene based on the correlation between the signal characteristics of each displayed second target object and the echo signal; the echo signal is obtained by the radar equipment capturing the micro-motion signal generated when the second target object is working; and a model updating module 1290, which is used to display the object mark of the newly deployed second target object in the corresponding area of the spatial layout diagram, and update the environmental model of the target scene.

[0179] In an exemplary embodiment, the second target object includes an electrically controlled object; the electrically controlled object refers to a device that can generate a micro-motion signal when working; the object identification module 1230 may also include: a first positioning unit 1231, which is used to locate the position of the electrically controlled object based on the echo signal, identify the position of the electrically controlled object in the target scene, and determine the corresponding area of the electrically controlled object in the spatial layout diagram; wherein the echo signal is obtained by the radar equipment capturing the micro-motion signal generated when the electrically controlled object is working.

[0180] In an exemplary embodiment, the object identification module 1230 may also include: a state identification unit 1232, which is used to identify that the state of the electronically controlled object is the on state based on the capture of the micro-motion signal generated by the radar equipment when the electronically controlled object is working, so as to mark the electronically controlled object in the on state in the corresponding area determined by the spatial layout diagram.

[0181] In an exemplary embodiment, the second target object includes a non-electrically controlled object; the non-electrically controlled object refers to a facility that cannot generate a micro-motion signal; the object identification module 1230 may also include: a second unit unit 1233, which is used to locate the position of the first target object interacting with the non-electrically controlled object based on the echo signal, and obtain the position of the first target object in the target scene; the echo signal is the radar device's transmitted signal reflected by the first target object; the position identification unit 1234 is used to identify the position of the non-electrically controlled object in the target scene based on the position of the first target object in the target scene, so as to determine the corresponding area of the non-electrically controlled object in the spatial layout diagram.

[0182] In an exemplary embodiment, the object recognition module 1230 may also include: a range determination unit 1235, which is used to determine the area range of the smart device in the target scene if it is detected that the state of the smart device changes from an off state to an on state; a first type recognition unit 1236, which is used to, within the determined area range, identify the device type of the smart device as the object type of the second target object if the radar device captures the micro-motion signal generated when the second target object is working, so as to mark the second target object according to the device type of the smart device in the spatial layout diagram.

[0183] In an exemplary embodiment, the object identification module 1230 may also include: a second type identification unit 1237, which is used to identify the object type of the second target object in response to a type marking operation for the second target object; or to identify the object type of the second target object in response to a recognition event triggered by the image acquisition device for the second target object.

[0184] In an exemplary embodiment, the device 1200 may also include: a trajectory information acquisition module 1320, used to obtain the trajectory information of the first target object in each space in the target scene, draw a trajectory map of each space and display it; the trajectory information is used to indicate the movement trajectory of the first target object in each space; an entrance and exit marking module 1340, used to mark the entrance and exit positions of each space on the trajectory map of each space; a layout map acquisition module, used to obtain a space layout map by matching the trajectory maps of each space with marked entrance and exit positions.

[0185] In an exemplary embodiment, the trajectory information acquisition module 1320 includes: a trajectory tracking unit 1321, which is used to track the trajectory of the first target object based on the echo signal to obtain the trajectory information of the first target object in each space; the echo signal is obtained by the radar device's transmission signal being reflected by the first target object.

[0186] In an exemplary embodiment, the entrance and exit marking module 1340 includes: a position determining unit 1341 for determining the position of the first target object in each space when the space opening and closing event is triggered in each space.

[0187] In an exemplary embodiment, the entrance and exit marking module 1340 may also include: an adjacent space determination unit 1342, which is used to determine two adjacent spaces associated with the connected position in response to a marking operation on the connected position; the connected position is used to indicate the position of the first target object when entering or exiting between the two adjacent spaces.

[0188] See also Figure 13 The embodiment of the present application also provides an environment model display device 1300, including but not limited to a model display module 1310 and a display update module 1330.

[0189] Among them, the model display module 1310 is used to display the environment model corresponding to the target scene; the state environment model includes the state of each first target object and each second target object in the state target scene; the state environment model is generated based on the spatial layout diagram of the state target scene and the state is based on the position of each state second target object in the target scene; the state spatial layout diagram is generated based on the trajectory of the state first target object in the state target scene and the relationship between each space in the state target scene.

[0190] The display update module 1330 is configured to display the updated state of the first target object and / or the second target object in the environment model if it is recognized that the state of each first target object and / or each second target object has changed.

[0191] It should be noted that the device for generating an environmental model provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when generating an environmental model. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device for generating an environmental model will be divided into different functional modules to complete all or part of the functions described above.

[0192] In addition, the embodiments of the device for generating an environment model and the method for generating an environment model provided in the above embodiments belong to the same concept, wherein the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0193] See also Figure 14 , Figure 12 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Figure 1 A user terminal 110 is shown in an implementation environment.

[0194] It should be noted that the electronic device is only an example adapted for this application and cannot be considered to provide any limitation on the scope of use of this application. The electronic device cannot be interpreted as needing to rely on or must have Figure 14 One or more components of the exemplary electronic device 1400 shown in FIG.

[0195] like Figure 14 As shown, the electronic device 1400 includes a memory 101, a storage controller 103, one or more ( Figure 14 The processor 105 (only one is shown), the peripheral interface 107, the radio frequency module 109, the positioning module 111, the camera module 113, the audio module 115, the touch screen 117 and the key module 119. These components communicate with each other via one or more communication buses / signal lines 121.

[0196] Among them, the memory 101 can be used to store computer programs and modules, such as the computer programs and modules corresponding to the environment model generation method and device in the exemplary embodiment of the present application. The processor 105 executes various functions and data processing by running the computer program stored in the memory 101, that is, completes the method of environment model generation.

[0197] The memory 101 is a carrier for resource storage and can be a random access memory, such as a high-speed random access memory, a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other solid-state memory. The storage method can be temporary storage or permanent storage.

[0198] The peripheral interface 107 may include at least one wired or wireless network interface, at least one serial-parallel conversion interface, at least one input / output interface, and at least one USB interface, etc., for coupling various external input / output devices to the memory 101 and the processor 105 to achieve communication with various external input / output devices.

[0199] The RF module 109 is used to transmit and receive electromagnetic waves, converting electromagnetic waves into electrical signals, thereby communicating with other devices via a communication network. The communication network includes a cellular telephone network, a wireless local area network, or a metropolitan area network. The above communication networks can use various communication standards, protocols, and technologies.

[0200] The positioning module 111 is used to obtain the current geographical location of the electronic device 1400. Examples of the positioning module 111 include, but are not limited to, a global positioning system (GPS) and positioning technologies based on a wireless local area network or a mobile communication network.

[0201] The camera module 113 is a camera that is used to take pictures or videos. The pictures or videos taken can be stored in the memory 101 and can also be sent to the host computer through the radio frequency module 109.

[0202] The audio module 115 provides an audio interface to the user, which may include one or more microphone interfaces, one or more speaker interfaces, and one or more headphone interfaces. The audio data is interacted with other devices through the audio interface. The audio data can be stored in the memory 101 and can also be sent through the radio frequency module 109.

[0203] The touch screen 117 provides an input / output interface between the electronic device 1100 and the user. Specifically, the user can use the touch screen 117 to perform input operations, such as clicks, touches, and slides, to cause the electronic device 1400 to respond to the input operations. The electronic device 1400 then displays output content, such as text, images, or videos, in any form or combination thereof, to the user via the touch screen 117.

[0204] The key module 119 includes at least one key to provide an interface for the user to input to the electronic device 1400. The user can press different keys to make the electronic device 1400 perform different functions. For example, the sound adjustment key allows the user to adjust the volume of the sound played by the electronic device 1400.

[0205] I understand. Figure 14 The structure shown is for illustration only. The electronic device 1400 may further include Figure 14 More or fewer components as shown, or with Figure 14 Different components are shown. Figure 14 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0206] See also Figure 15 In an embodiment of the present application, an electronic device 4000 is provided, which may include: a smart phone, a tablet computer, etc.

[0207] exist Figure 15 In the embodiment, the electronic device 4000 includes at least one processor 4001, at least one communication bus 4002 and at least one memory 4003.

[0208] The processor 4001 and the memory 4003 are connected, for example, via a communication bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0209] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0210] The communication bus 4002 may include a path for transmitting information between the above components. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0211] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0212] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002 .

[0213] When the computer program is executed by the processor 4001 , the method for generating an environment model and the method for displaying an environment model in the above-mentioned embodiments are implemented.

[0214] In addition, a storage medium is provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the method for generating an environmental model and the method for displaying an environmental model in the above-mentioned embodiments are implemented.

[0215] In an embodiment of the present application, a computer program product is provided, comprising a computer program stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to perform the method for generating an environment model and the method for displaying an environment model in the above-described embodiments.

[0216] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0217] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating an environment model, characterized in that: The method comprises: Displaying a spatial layout diagram corresponding to the target scene; the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and each space in the target scene; Identifying each second target object deployed in the target scene, and obtaining a position of each identified second target object in the target scene; generating an environment model of the target scene according to the position of each second target object in the target scene and a spatial layout diagram corresponding to the target scene; the environment model is used to describe the state of the first target object and / or the second target object in the target scene; The step of obtaining the spatial layout diagram includes: Obtaining trajectory information of the first target object in each space in the target scene, drawing and displaying a trajectory diagram of each space; the trajectory information is used to indicate the movement trajectory of the first target object in each space; In response to a space opening and closing event triggered in each space, respectively determining a position of the first target object in each space when the space opening and closing event is triggered; According to the determined positions, the entrance and exit positions of each space are marked in the corresponding areas in the trajectory diagram of each space; and the spatial layout diagram is obtained by matching the trajectory diagrams of each space with the marked entrance and exit positions.

2. The method according to claim 1, wherein Generating the environment model of the target scene according to the position of each second target object in the target scene and the spatial layout diagram corresponding to the target scene includes: determining, according to the position of each second target object in the target scene, a corresponding area of each second target object in the spatial layout diagram; The object mark of each second target object is displayed in a corresponding area in the spatial layout diagram to generate an environmental model of the target scene; the object mark of the second target object is used to indicate the status of the second target object.

3. The method according to claim 2, wherein After displaying the object mark of each second target object in the corresponding area of the spatial layout diagram and generating the environment model of the target scene, the method further includes: identifying a second target object newly deployed in the target scene based on the displayed correlation between each second target object and the signal characteristics of the echo signal, wherein the echo signal is obtained by the radar device capturing a micro-motion signal generated when the second target object is operating; The object mark of the newly deployed second target object is displayed in a corresponding area in the spatial layout diagram, and the environment model of the target scene is updated.

4. The method according to claim 1, wherein The second target object includes an electronically controlled object; the electronically controlled object refers to a device that can generate a micro motion signal when in operation; The identifying each second target object deployed in the target scene includes: Positioning the electrically controlled object based on the echo signal, identifying the position of the electrically controlled object in the target scene, and determining a corresponding area of the electrically controlled object in the spatial layout diagram; The echo signal is obtained by the radar device capturing the micro-motion signal generated when the electronically controlled object is working.

5. The method according to claim 4, wherein The identifying of each second target object deployed in the target scene further includes: Based on the radar device capturing the micro-motion signal generated when the electrically controlled object is working, it is identified that the state of the electrically controlled object is on, so that the electrically controlled object in the on state is marked in the corresponding area determined by the spatial layout diagram.

6. The method according to claim 1, wherein The second target object includes a non-electrically controlled object; the non-electrically controlled object refers to a facility that cannot generate a micro-motion signal; The identifying each second target object deployed in the target scene includes: Locating the position of the first target object interacting with the non-electrically controlled object based on the echo signal, to obtain the position of the first target object in the target scene; the echo signal is obtained by reflecting the transmission signal of the radar device from the first target object; According to the position of the first target object in the target scene, the position of the non-electrically controlled object in the target scene is identified, so as to determine the corresponding area of the non-electrically controlled object in the spatial layout diagram.

7. The method according to claim 1, wherein The second target object includes a smart device; and the identifying each second target object deployed in the target scene includes: If it is detected that the state of the smart device changes from the off state to the on state, determining the area range of the smart device in the target scene; Within the determined area, if the radar device captures the micro-motion signal generated when the second target object is working, the device type of the smart device is identified as the object type of the second target object, so as to mark the second target object according to the device type of the smart device in the spatial layout diagram.

8. The method according to claim 1, wherein The obtaining of trajectory information of the first target object in each space in the target scene includes: The trajectory of the first target object is tracked based on the echo signal to obtain the trajectory information of the first target object in each of the spaces; the echo signal is obtained by reflecting the transmission signal of the radar device through the first target object.

9. A method for displaying an environment model, characterized in that: The method comprises: Displaying an environment model corresponding to a target scene; the environment model includes the states of each first target object and / or each second target object in the target scene; the environment model is generated based on a spatial layout diagram of the target scene and the positions of each second target object in the target scene; the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and each space in the target scene; If a change in the state of each of the first target objects and / or each of the second target objects is identified, displaying the updated state of the first target object and / or the second target object in the environment model; The step of obtaining the spatial layout diagram includes: Obtaining trajectory information of the first target object in each space in the target scene, drawing and displaying a trajectory diagram of each space; the trajectory information is used to indicate the movement trajectory of the first target object in each space; In response to a space opening and closing event triggered in each space, respectively determining a position of the first target object in each space when the space opening and closing event is triggered; According to the determined positions, the entrance and exit positions of each space are marked in the corresponding areas in the trajectory diagram of each space; and the spatial layout diagram is obtained by matching the trajectory diagrams of each space with the marked entrance and exit positions.

10. A device for generating an environment model, characterized in that: The device comprises: A layout diagram display module is used to display a spatial layout diagram corresponding to a target scene; the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and each space in the target scene; an object recognition module, configured to recognize each second target object deployed in the target scene, and obtain a position of each recognized second target object in the target scene; a model generation module, configured to generate an environment model of the target scene based on the position of each second target object in the target scene and a spatial layout diagram corresponding to the target scene; the environment model is used to describe the state of the first target object and / or the second target object in the target scene; a trajectory information acquisition module, configured to acquire trajectory information of the first target object in each space in the target scene, draw and display trajectory diagrams of each space; the trajectory information is used to indicate the movement trajectory of the first target object in each space; an entrance and exit marking module, configured to respond to a space opening and closing event triggered in each space and respectively determine a position of the first target object in each space when the space opening and closing event is triggered; According to the determined positions, the entrance and exit positions of each space are marked in the corresponding areas of the trajectory diagram of each space; The layout map acquisition module is used to obtain a space layout map by matching the trajectory maps of each space with marked entrance and exit positions.

11. A display device for an environmental model, characterized in that: The device comprises: A model display module is configured to display an environment model corresponding to a target scene; the environment model includes the status of each first target object and / or each second target object in the target scene; the environment model is generated based on a spatial layout diagram of the target scene and the position of each second target object in the target scene; the spatial layout diagram is generated based on the relationship between the trajectory of the first target object in the target scene and each space in the target scene; a display update module, configured to display the updated state of the first target object and / or the second target object in the environment model if it is recognized that the state of each of the first target objects and / or each of the second target objects has changed; a trajectory information acquisition module, configured to acquire trajectory information of the first target object in each space in the target scene, draw and display a trajectory diagram of each space; the trajectory information is used to indicate the movement trajectory of the first target object in each space; an entrance and exit marking module, configured to respond to a space opening and closing event triggered in each space and respectively determine a position of the first target object in each space when the space opening and closing event is triggered; According to the determined positions, the entrance and exit positions of each space are marked in the corresponding areas of the trajectory diagram of each space; The layout map acquisition module is used to obtain the space layout map by matching the trajectory map of each space with the entrance and exit positions marked.

12. An electronic device, characterized in that: include: At least one processor, at least one memory, wherein A computer program is stored in the memory, and the processor reads the computer program in the memory; When the computer program is executed by the processor, the method according to any one of claims 1 to 9 is implemented.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images