Virtual object layout method and virtual object layout information acquisition method and device

Abstract

The invention relates to a virtual object layout method, a virtual object layout information acquisition method and device, electronic equipment, a storage medium and a computer program product, and relates to the technical field of Internet of Things. The virtual object layout efficiency can be improved. The method comprises the steps that a virtual object layout instruction is received, the instruction is responded, layout information of a virtual object in a virtual space is obtained, the layout information is obtained according to an image of the entity space containing the entity object and a track surrounding the entity object, the virtual space corresponds to the entity space, and the virtual object corresponds to the entity object; and displaying the virtual object in the virtual space according to the layout information of the virtual object in the virtual space.

Description

Technical Field

[0001] The present application relates to the technical field of the Internet of Things, and particularly to a method for laying out virtual objects, a method for obtaining virtual object layout information, a device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] With the development of Internet of Things technology, some application programs provide a function of laying out virtual objects for users on an interface.

[0003] In the virtual object layout method provided by the current technology, the user needs to select on the interface by himself / herself and place the virtual objects at corresponding positions in the virtual space through operations such as dragging, resulting in the problem of low efficiency of virtual object layout. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method for laying out virtual objects, a method for obtaining virtual object layout information, a device, an electronic device, a storage medium, and a computer program product.

[0005] In a first aspect, the present application provides a method for laying out virtual objects. The method includes:

[0006] Receiving a virtual object layout instruction;

[0007] In response to the virtual object layout instruction, obtaining layout information of the virtual object in the virtual space; the layout information is obtained according to an image of an entity space including an entity object and a trajectory around the entity object; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object;

[0008] Displaying the virtual object in the virtual space according to the layout information.

[0009] In one embodiment, the obtaining layout information of the virtual object in the virtual space includes: obtaining first contour information and obtaining second contour information; the first contour information is obtained according to the image; the second contour information is obtained according to the trajectory; if the first contour information matches the second contour information, then according to the identification information and layout position information of the entity object, obtaining the layout information of the virtual object in the virtual space; the identification information is obtained according to the image; the layout position information is obtained according to the trajectory.

[0010] In one embodiment, the obtaining the layout information of the virtual object in the virtual space according to the identification information and layout position information of the entity object includes: determining the corresponding virtual object in the virtual object library according to the identification information; obtaining the layout information of the virtual object in the virtual space according to the virtual object and the layout position information.

[0011] In one embodiment, the first contour information includes the first contour information of a plurality of entity objects; the second contour information includes the second contour information of the plurality of entity objects; the method further includes: determining the first contour information that does not meet the preset contour similarity condition among the first contour information of the plurality of entity objects; performing a first matching on the first contour information that does not meet the preset contour similarity condition and the second contour information of the plurality of entity objects; determining the positional relationship between the unmatched first contour information and the matched first contour information among the first contour information of the plurality of entity objects; and performing a second matching on the unmatched first contour information and the unmatched second contour information among the second contour information of the plurality of entity objects according to the positional relationship.

[0012] In one embodiment, the determining the first contour information that does not meet the preset contour similarity condition among the first contour information of the plurality of entity objects includes: clustering the first contour information of the plurality of entity objects according to contour similarity to obtain a plurality of first contour information groups of different categories; and determining the first contour information that does not meet the preset contour similarity condition among the first contour information of the plurality of entity objects according to the number of first contour information included in each of the plurality of first contour information groups.

[0013] In one embodiment, the obtaining the first contour information includes: inputting the image into a trained semantic segmentation model; obtaining the image region of the entity object in the image output by the semantic segmentation model; and obtaining the first contour information according to the image region. After inputting the image into the trained semantic segmentation model, the method further includes: obtaining the identification information of the entity object included in the image output by the semantic segmentation model.

[0014] In one embodiment, the method further includes: obtaining image training samples of various types of entity objects and obtaining the calibration information of the image training samples; the calibration information includes information for calibrating the image region where the entity object is located in the image training sample and information for calibrating the identification information of the entity object in the image training sample; and training the to-be-trained semantic segmentation model according to the image training samples and the calibration information.

[0015] In a second aspect, the present application further provides a method for obtaining virtual object layout information. The method includes:

[0016] obtaining first contour information; the first contour information is obtained according to an image of an entity space including entity objects;

[0017] obtaining a trajectory around the entity object;

[0018] Obtain the layout information of the virtual object in the virtual space according to the first contour information and the trajectory; the virtual space corresponds to the physical space; the virtual object corresponds to the physical object.

[0019] In one embodiment, the obtaining the layout information of the virtual object in the virtual space according to the first contour information and the trajectory includes: obtaining second contour information according to the trajectory; if the first contour information matches the second contour information, obtaining the layout position information of the physical object according to the trajectory, and obtaining the layout information of the virtual object in the virtual space according to the layout position information and the obtained identification information of the physical object; the identification information is obtained according to the image.

[0020] In one embodiment, the first contour information includes the first contour information of multiple physical objects; the second contour information includes the second contour information of multiple physical objects; the method further includes: determining the first contour information that does not meet the preset contour similarity condition among the first contour information of the multiple physical objects; performing a first match between the first contour information that does not meet the preset contour similarity condition and the second contour information of the multiple physical objects; determining the positional relationship between the unmatched first contour information and the matched first contour information among the first contour information of the multiple physical objects; and performing a second match between the unmatched first contour information and the unmatched second contour information among the second contour information of the multiple physical objects according to the positional relationship.

[0021] In one embodiment, the determining the first contour information that does not meet the preset contour similarity condition among the first contour information of the multiple physical objects includes: clustering the first contour information of the multiple physical objects according to contour similarity to obtain multiple first contour information groups of different categories; and determining the first contour information that does not meet the preset contour similarity condition among the first contour information of the multiple physical objects according to the number of first contour information included in each of the multiple first contour information groups.

[0022] In a third aspect, the present application further provides a virtual object layout device. The device includes:

[0023] An instruction receiving module, configured to receive a virtual object layout instruction;

[0024] A layout acquisition module, configured to obtain layout information of a virtual object in a virtual space in response to the virtual object layout instruction; the layout information is obtained based on an image of an entity space including an entity object and a trajectory around the entity object; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object;

[0025] An object display module, configured to display the virtual object in the virtual space according to the layout information.

[0026] In a fourth aspect, the present application further provides a virtual object layout information acquisition device. The device includes:

[0027] An information acquisition module, configured to acquire first contour information; the first contour information is obtained based on an image of an entity space including an entity object;

[0028] A trajectory acquisition module, configured to acquire a trajectory around the entity object;

[0029] A layout obtaining module, configured to obtain layout information of a virtual object in a virtual space according to the first contour information and the trajectory; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object.

[0030] In a fifth aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Receive a virtual object layout instruction; in response to the virtual object layout instruction, obtain layout information of a virtual object in a virtual space; the layout information is obtained based on an image of an entity space including an entity object and a trajectory around the entity object; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object; display the virtual object in the virtual space according to the layout information.

[0032] In a sixth aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Obtain first contour information; the first contour information is obtained based on an image of an entity space including an entity object; obtain a trajectory around the entity object; obtain layout information of a virtual object in a virtual space according to the first contour information and the trajectory; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object.

[0034] In a seventh aspect, the present application also provides a computer-readable storage medium. On the computer-readable storage medium, there is stored a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0035] Receive a virtual object layout instruction; in response to the virtual object layout instruction, obtain layout information of the virtual object in the virtual space; the layout information is obtained based on an image of an entity space including an entity object and a trajectory around the entity object; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object; display the virtual object in the virtual space according to the layout information.

[0036] In an eighth aspect, the present application also provides a computer-readable storage medium. On the computer-readable storage medium, there is stored a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0037] Obtain first contour information; the first contour information is obtained based on an image of an entity space including an entity object; obtain a trajectory around the entity object; obtain layout information of the virtual object in the virtual space according to the first contour information and the trajectory; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object.

[0038] In a ninth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Receive a virtual object layout instruction; in response to the virtual object layout instruction, obtain layout information of the virtual object in the virtual space; the layout information is obtained based on an image of an entity space including an entity object and a trajectory around the entity object; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object; display the virtual object in the virtual space according to the layout information.

[0040] In a tenth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0041] Obtain first contour information; the first contour information is obtained based on an image of an entity space including an entity object; obtain a trajectory around the entity object; obtain layout information of the virtual object in the virtual space according to the first contour information and the trajectory; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object.

[0042] The above virtual object layout method, virtual object layout information acquisition method, device, electronic device, storage medium, and computer program product receive a virtual object layout instruction. In response to this instruction, the layout information of the virtual object in the virtual space is obtained. This layout information is obtained based on the image of the physical space containing the physical object and the trajectory around the physical object. The virtual space corresponds to the physical space, and the virtual object corresponds to the physical object. According to the layout information of the virtual object in the virtual space, the virtual object is displayed in the virtual space. This solution can automatically obtain the layout information of the virtual object in the virtual space based on the image of the physical space containing the physical object and the trajectory around the physical object when receiving the user's virtual object layout instruction, and display the virtual object in the virtual space according to this layout information. Without the user having to perform cumbersome operations such as selecting and dragging the virtual object, it can automatically layout the virtual object in the virtual space for the user, improving the efficiency of virtual object layout and also enhancing the convenience of the user in laying out virtual objects. Description of the Drawings

[0043] Figure 1 It is an application environment diagram of the related method in an embodiment of this application;

[0044] Figure 2 It is a schematic flowchart of the virtual object layout method in an embodiment of this application;

[0045] Figure 3 It is a schematic diagram of a virtual object in an embodiment of this application;

[0046] Figure 4 It is a schematic diagram of virtual object layout in an embodiment of this application;

[0047] Figure 5 It is a schematic diagram of contour information in an embodiment of this application;

[0048] Figure 6 It is a schematic flowchart of the steps of contour matching processing in an embodiment of this application;

[0049] Figure 7 It is a schematic flowchart of the virtual object layout information acquisition method in an embodiment of this application;

[0050] Figure 8 It is a structural block diagram of the virtual object layout device in an embodiment of this application;

[0051] Figure 9 It is a structural block diagram of the virtual object layout information acquisition device in an embodiment of this application;

[0052] Figure 10 It is an internal structural diagram of the electronic device in an embodiment of this application. Detailed Embodiments

[0053] In order to make the objectives, technical solutions, and advantages of the present application more clear and understandable, the present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The virtual object layout method and the virtual object layout information acquisition method of the present application can be applied to an application environment as Figure 1 shown. This application environment can be a smart home system, which can include: a cloud / cloud server, a router, a gateway, user terminals (such as mobile phones, tablets), and multiple sub-devices (such as cameras, millimeter-wave radars, sensors, switches), etc. In the smart home system, the sub-devices can be connected to the gateway through ZIGBEE / Bluetooth / WiFi, and the gateway and the user terminal can be respectively connected to the router through WiFi. In addition, the user terminal can also establish a network connection with the cloud / cloud server through 2G / 3G / 4G / 5G, WiFi, etc., so as to obtain the data sent by the cloud / cloud server.

[0055] For the smart home system, it can adopt the following system configuration method: a corresponding application program is installed on the user terminal. First, at least one gateway device can be added on the application program. When adding other sub-devices (such as cameras, millimeter-wave radars, sensors, switches), the add sub-device button on the interface of the application program can be clicked, and the gateway device to which the sub-device needs to be connected can be selected in the add interface, so that the sub-device can be added to the ZIGBEE network corresponding to the gateway, and then together with other added sub-devices and the gateway, a ZIGBEE local area network is formed. When the application program, the router, and the gateway are in the same local area network, the application program can interact with the gateway and the sub-devices connected to the gateway through the local area network path; when the application program, the router, and the gateway are not in the same local area network, the application program can interact with the gateway and the sub-devices connected to the gateway through the wide area network path.

[0056] In one embodiment, as Figure 2 shown, the present application provides a virtual object layout method, which can be executed by a user terminal as Figure 1 shown. This method can include the following steps:

[0057] Step S201, receiving a virtual object layout instruction.

[0058] Among them, the virtual object layout instruction refers to an instruction used to direct the user terminal to perform virtual object layout. This instruction can be triggered by the user on the user terminal. The user terminal can provide a button on the interface of the application for triggering the virtual object layout instruction. When the user clicks this button, the virtual object layout instruction is triggered, and the user terminal receives the virtual object layout instruction. A virtual object refers to an object that can be displayed on the interface of the application. In the application, this virtual object can be presented in the form of a sticker. Taking the scenario of a smart home system as an example, as Figure 3 shown, the virtual object can be represented as stickers of various furniture (such as doors, curtains, sofas, TVs).

[0059] Among them, the virtual object corresponds to the physical object. A virtual object can correspond to a physical object or a category of physical objects. A physical object refers to an object in the physical space. For example, a virtual object can be represented as a sticker of a single door or a sticker of a category of doors. Among them, the physical space corresponds to the virtual space. The physical space is the space in the real world where physical objects can be placed. The virtual space can be the space of the interface of the application installed on the user terminal and can be used for laying out virtual objects. As an example, the physical space can be the living room in a residence, and the virtual space can be the virtual living room corresponding to this living room displayed in the application.

[0060] Thus, in this step, the virtual object layout instruction can be an instruction for the user to direct the user terminal to automatically display virtual objects in the virtual space according to the relevant information of the physical objects in the physical space, so that the layout of the virtual objects in the virtual space corresponds to the layout of the physical objects in the physical space.

[0061] Step S202: In response to the virtual object layout instruction, obtain the layout information of the virtual object in the virtual space.

[0062] In this step, the user terminal obtains the layout information of the virtual object in the virtual space in response to the virtual object layout instruction triggered by the user on the interface of the application. Among them, the layout information of the virtual object in the virtual space may include the layout position information of the virtual object in the virtual space. The layout position information is obtained based on the image of the physical space containing the physical object and the trajectory around the physical object. The image of the physical space containing the physical object can be obtained by photographing the physical space containing the physical object. In the scenario of a smart home system, the image of the living room (corresponding to the physical space) containing furniture (corresponding to the physical object) can be obtained by the camera of the smart home system. The trajectory around the physical object can be a trajectory formed by the user moving around the contour of the physical object for one circle. The trajectory can be detected by the millimeter-wave radar when the user moves around the contour of the physical object for one circle. In the scenario of a smart home system, the trajectory around the physical object can be detected by the millimeter-wave radar of the smart home system. That is, when the user needs the user terminal to layout the virtual object in the virtual space according to the physical object in the physical space, the user terminal can obtain the layout information of the virtual object in the virtual space based on the image obtained by photographing the physical space containing the physical object and the trajectory formed by moving around the contour of the physical object. The layout information may include the layout position information of the virtual object in the virtual space, and the layout position information may include information such as the layout coordinates and orientation of the virtual object in the virtual space.

[0063] Step S203: Display the virtual object in the virtual space according to the layout information.

[0064] In this step, the user terminal can display the virtual object in the virtual space according to the layout position information of the virtual object in the virtual space in the layout information. The number of virtual objects can be one or multiple. That is, by photographing multiple physical objects in the physical space and obtaining the trajectories around each physical object, the layout information of each virtual object in the virtual space can be obtained, and each virtual object can be displayed in the virtual space according to the layout information of each virtual object in the virtual space. Thus, the layout of each virtual object corresponding to each physical object in the virtual space can be completed, improving the layout efficiency and convenience. For example Figure 4As shown, taking the scenario of a smart home system as an example, an image of a living room containing various furniture can be captured by a camera of the smart home system, and trajectories around various furniture in the living room can be obtained through a millimeter-wave radar of the smart home system. Thus, layout information of each furniture sticker (objects 1 to 5) in the virtual living room can be obtained. According to the layout information of each furniture sticker in the virtual living room, each furniture sticker is displayed in the virtual living room, and the position where the millimeter-wave radar is located can also be indicated in the virtual living room. Therefore, compared with the traditional technology, the manual configuration operation can be omitted. The user can move around the contours of each furniture in the living room for one circle to obtain the trajectory by the millimeter-wave radar, and combine the image of the living room containing various furniture captured by the camera to obtain the furniture stickers corresponding to each furniture and their layout information. Based on this, these furniture stickers are automatically laid out in the virtual living room, improving the efficiency and convenience of laying out each furniture sticker in the application program of the smart home system.

[0065] In the virtual object layout method of this embodiment, a virtual object layout instruction is received. In response to this instruction, layout information of the virtual object in the virtual space is obtained. This layout information is obtained based on an image of an entity space containing entity objects and a trajectory around the entity objects. The virtual space corresponds to the entity space, and the virtual object corresponds to the entity object. According to the layout information of the virtual object in the virtual space, the virtual object is displayed in the virtual space. This solution can automatically obtain the layout information of the virtual object in the virtual space obtained based on an image of an entity space containing entity objects and a trajectory around the entity objects when receiving the user's virtual object layout instruction, and display the virtual object in the virtual space according to this layout information. Without the user performing cumbersome operations such as selecting and dragging the virtual object, the virtual object can be automatically laid out in the virtual space for the user, improving the virtual object layout efficiency and also improving the convenience of the user laying out the virtual object.

[0066] Regarding the acquisition of the layout information of the virtual object in the virtual space, in one embodiment, the acquisition of the layout information of the virtual object in the virtual space in step S202 may include:

[0067] Obtain first contour information and obtain second contour information; if the first contour information matches the second contour information, then according to the identification information and layout position information of the entity object, obtain the layout information of the virtual object in the virtual space.

[0068] This embodiment may involve the matching of contour information and the acquisition of layout information when the matching occurs. Among them, the matching of the contour information in this embodiment refers to the matching of the first contour information and the second contour information. The first contour information can be obtained from an image of an entity space containing an entity object, and the second contour information can be obtained from a trajectory surrounding the entity object. The first and second contour information are used to depict the contour of the entity object. The first contour information of the entity object can be extracted from the image of the entity space containing the entity object, and the trajectory formed by moving around the entity object once can be used as the second contour information. As Figure 5 shown, the first contour information and the second contour information of each entity object can be obtained. The first contour information and the second contour information of objects 3 to 5 are not shown in Figure 5 . Thus, the first contour information can be matched with the second contour information. If the first contour information matches the second contour information, the layout information of the virtual object in the virtual space can be obtained according to the identification information and the layout position information of the corresponding entity object. Among them, the identification information of the entity object refers to the information used to identify the entity object. For example, it can be the category information (such as sofa, table, chair, etc.) to which the entity object belongs. The identification information of the entity object can be obtained from the image of the entity space containing the entity object described above. For example, the identification information of the entity object can be obtained by recognizing the entity object contained in the image, etc.; the layout position information can include information such as the coordinates and orientation of the entity object. The layout position information can be obtained from the trajectory surrounding the entity object described above. For example, the coordinates, orientation, etc. of the entity object relative to the millimeter-wave radar can be determined according to the trajectory and used as the layout position information. Taking Figure 5 as an example, the first contour information of object 1 can be matched with each second contour information. If the first contour information matches a second contour information, the identification information of object 1 obtained from the image described above can be obtained, and the layout position information of object 1 obtained from the trajectory described above can be obtained. According to the identification information and the layout position information, the layout information of the virtual object in the virtual space can be obtained. Based on the contour matching method, this embodiment can accurately and efficiently identify the virtual object and its layout information.

[0069] As one of the embodiments, obtaining the layout information of the virtual object in the virtual space according to the identification information and the layout position information of the entity object in the above embodiment may include:

[0070] Determine the corresponding virtual object in the virtual object library according to the identification information; obtain the layout information of the virtual object in the virtual space according to the virtual object and the layout position information.

[0071] In this embodiment, the virtual object library can be pre-established in the application of the user terminal. The virtual object library can include multiple virtual objects, and each virtual object can correspond to an identification information, such as Figure 3 The virtual object library in the scene of the smart home system is shown, which can be represented in the form of a furniture sticker library. Each furniture sticker in the furniture sticker library can correspond to an identification information, and the identification information can be used to indicate the category to which the furniture sticker belongs (such as bed, sofa, etc.). Therefore, the virtual object corresponding to the identification information in the virtual object library can be determined according to the identification information, and the virtual object and layout position information can be used as the layout information of the virtual object in the virtual space. The solution of this embodiment can accurately and quickly match the corresponding virtual object in the virtual object library of the application based on the identification information, providing a basis for accurately and conveniently arranging virtual objects in the virtual space.

[0072] As one of the embodiments, the matching process of the first profile information and the second profile information in the above embodiment can be performed through the following steps: Figure 6 As shown, the method of the present application also includes the following steps:

[0073] Step S601: determining first contour information that does not meet a preset contour similarity condition among first contour information of a plurality of entity objects.

[0074] Step S602: first matching is performed between first contour information that does not meet a preset contour similarity condition and second contour information of a plurality of entity objects.

[0075] Step S603 : determining a positional relationship between unmatched first contour information among the first contour information of the plurality of entity objects and matched first contour information among the first contour information of the plurality of entity objects.

[0076] Step S604: performing a second matching on the unmatched first contour information in the first contour information of the plurality of entity objects and the unmatched second contour information in the second contour information of the plurality of entity objects according to the position relationship.

[0077] In this embodiment, the first contour information includes the first contour information of multiple entity objects, and the second contour information includes the second contour information of multiple entity objects. An image of the entity space containing each entity object can be captured, and the trajectory around each entity object can be obtained at one time to match the contour information, thereby improving the matching efficiency. Figure 5, the first contour information may include the first contour information of object 1 to object 5, and the second contour information may include the second contour information of object 1 to object 5. Thus, in step S601, the first contour information that does not meet the preset contour similarity condition among the first contour information of multiple entity objects can be determined. The preset contour similarity condition may be the same as or close to the contours of other entity objects. Not meeting the preset contour similarity condition means being different from and not close to the contours of other entity objects. Then in step S602, the first contour information that does not meet the preset contour similarity condition is matched with the second contour information of multiple entity objects (denoted as the first match for distinction from the subsequent matching process), that is, the first contour information of the entity object that is different from and not close to the contours of other entity objects is preferentially matched with the second contour information of multiple entity objects, so that the matching first contour information and second contour information can be accurately determined. Then, among the first contour information of multiple entity objects, some first contour information is not matched, and some first contour information is already matched, which are respectively denoted as the unmatched first contour information among the first contour information of multiple entity objects and the matched first contour information among the first contour information of multiple entity objects. Then, the position relationship (such as relative distance relationship, relative orientation relationship, etc.) between the matched first contour information and the unmatched first contour information is used to match the unmatched first contour information. In this regard, in steps S603 and S604, the position relationship (such as relative distance relationship, relative orientation relationship, etc.) between the unmatched first contour information among the first contour information of multiple entity objects and the matched first contour information among the first contour information of multiple entity objects is determined. According to this position relationship, the unmatched first contour information among the first contour information of multiple entity objects is matched with the unmatched second contour information among the second contour information of multiple entity objects (denoted as the second match). Thus, the first contour information that does not meet the preset contour similarity condition can be preferentially matched, and the matching first contour information and second contour information can be accurately determined. Then, a reference of the position relationship is provided for the first contour information that meets the preset contour similarity condition, and the first contour information that meets the preset contour similarity condition is matched according to the reference of the position relationship, so that the second contour information that matches the first contour information with similar or identical contours can also be accurately determined.

[0078] As one of the embodiments, in step S601 of the above embodiment, determining the first contour information that does not meet the preset contour similarity condition among the first contour information of multiple entity objects may include:

[0079] Cluster the first contour information of multiple entity objects according to contour similarity to obtain multiple groups of first contour information of different categories; determine the first contour information that does not meet the preset contour similarity condition among the first contour information of multiple entity objects according to the number of first contour information included in each of the multiple groups of first contour information.

[0080] In this embodiment, first, according to contour similarity, the first contour information of multiple entity objects can be clustered, so that those with the same or similar contours can be grouped together. For example, the first contour information of multiple entity objects can be clustered into 5 categories, forming 5 groups of first contour information , each group of first contour information corresponding to a different category. Each group of first contour information can include one or more first contour information. Then, according to the number of first contour information included in each of the multiple groups of first contour information, determine the first contour information that does not meet the preset contour similarity condition among the first contour information of multiple entity objects. As an example, the group of first contour information that only contains one first contour information in the 5 groups of first contour information can be determined as the first contour information that does not meet the preset contour similarity condition. Then, this group of first contour information can be preferentially and first matched with the second contour information of multiple entity objects. By doing so, the entity object with a unique contour can be determined first. Among them, if the group of first contour information that meets the preset contour similarity condition is matched with multiple second contour information, it will match successfully with multiple second contour information, and in this way, it is impossible to determine which second contour information this group of first contour information is matched with. At this time, the position relationship between the unmatched group of first contour information that meets the preset contour similarity condition and the already matched first contour information that does not meet the preset contour similarity condition can be used to determine, combined with Figure 5, the outline of Object 1 is unique therein. Therefore, the first outline information of Object 1 is preferentially matched with each second outline information, and it is easy to match the second outline information corresponding to Object 1. When the first outline information of Object 2 is matched with each second outline information, since the outline of Object 2 is very similar to the outline of, for example, Object 3, the first outline information of Object 2 may be successfully matched with the second outline information of Object 2 and the second outline information of Object 3. However, according to the positional relationship, the outline of Object 2 is closer to the outline of Object 1 than the outline of Object 3, and the position of Object 2 is closer to Object 1 than the position of Object 3. Therefore, the first outline information of Object 2 can be matched with the second outline information of Object 2. Thus, the first and second outline information of each entity object can be accurately and quickly matched. After the outline information is successfully matched, the layout information of each virtual object in the virtual space can be obtained according to the identification information and the layout position information, and each virtual object can be displayed in the virtual space according to the layout information.

[0081] As one of the embodiments, obtaining the first outline information in the above embodiment may include the following steps:

[0082] Input the image into a trained semantic segmentation model; obtain the image regions of the entity objects in the image output by the semantic segmentation model; and obtain the first outline information according to the image regions.

[0083] In this embodiment, the semantic segmentation model can be pre-trained. The semantic segmentation model can adopt a U-NET neural network. Thus, in the application, the image of the entity space containing the entity objects can be input into the trained semantic segmentation model to obtain the image regions of the entity objects in the image output by the semantic segmentation model. The image regions may include all the pixels occupied by the entity objects in the image. Then, the first outline information of the entity objects can be extracted according to the image regions.

[0084] In one embodiment, after inputting the image into the trained semantic segmentation model, the following steps may further be included: obtaining the identification information of the entity objects included in the image output by the semantic segmentation model.

[0085] In this embodiment, the semantic segmentation model can be used to complete the recognition of the image regions of the entity objects in the image and the recognition of the identification information of the entity objects, and can also be used to determine information such as the coordinates of the entity objects in the image.

[0086] As one of the embodiments, the above-mentioned semantic segmentation model can be trained through the following steps. The method of this application may further include the following steps:

[0087] Obtain image training samples of various entity objects, and obtain the calibration information of the image training samples; train the semantic segmentation model to be trained according to the image training samples and the calibration information.

[0088] In this embodiment, image training samples containing various entity objects can be collected. In the scenario of a smart home system, a camera of the smart home system can be used to collect many images containing entity objects such as sofas, tables, chairs, and coffee tables as training samples. Then, the image regions where these entity objects are located in the image training samples can be calibrated, and the identification information of the entity objects in the image training samples can be calibrated to obtain the calibration information of the image training samples. The calibration information can include the information calibrated for the image regions where the entity objects are located in the image training samples, and the information calibrated for the identification information of the entity objects in the image training samples. For example, the pixels occupied by various furniture can be calibrated in the image training samples, and identification information (1, 2, 3, 4, 5, etc.) can be calibrated for the entity objects in the image training samples. Then, the semantic segmentation model to be trained is trained according to the image training samples and the calibration information. As a model training method, the image training samples can be divided into a training set and a test set according to a ratio such as 7:3. Set initial network model parameters for the semantic segmentation model to be trained. After passing each image training sample in the training set through the semantic segmentation model to be trained, obtain the prediction results of the identification information of each pixel, and adjust the semantic segmentation model to be trained according to the difference between the prediction results of the identification information of each pixel and the calibration information until the semantic segmentation model meets certain requirements when tested in the test set, such as the pixel segmentation accuracy in the test set is greater than a preset accuracy threshold, etc., thereby obtaining a trained semantic segmentation model.

[0089] In one embodiment, a method for obtaining virtual object layout information is further provided. This method can be applied to any one of the millimeter-wave radar, camera, and gateway in the smart home system as shown in Figure 1 shown, as shown in Figure 7 shown, this method may include the following steps:

[0090] Step S701, obtain the first contour information.

[0091] Step S702, obtain the trajectory around the entity object.

[0092] Step S703, obtain the layout information of the virtual object in the virtual space according to the first contour information and the trajectory.

[0093] In this embodiment, the smart home system may include a camera and a millimeter-wave radar. The camera can be used to capture an image of the physical space containing physical objects, obtain first contour information based on the image, and also obtain the identification information of the physical objects contained in the image. The camera can input the image into a trained semantic segmentation model to obtain information such as the first contour information and the identification information of the physical objects output by the trained semantic segmentation model. The camera can transmit the first contour information, the identification information of the physical objects, and other information to the millimeter-wave radar, and the millimeter-wave radar can obtain the first contour information, the identification information of the physical objects, and other information transmitted by the camera. The millimeter-wave radar can also detect the trajectory of the user's movement, obtain the trajectory of the user moving around a physical object for one circle. As an example, the user can move around the contour of a physical object for one circle, and the millimeter-wave radar can record its trajectory. After all the physical objects in the physical space are moved around by the user for one circle, each physical object can correspond to a trajectory. Thus, the millimeter-wave radar can combine the first contour information and the trajectory to obtain the layout information of the virtual object in the virtual space. The layout information of the virtual object in the virtual space can be used to display the virtual object in the virtual space. For example, the millimeter-wave radar can send the layout information of the virtual object in the virtual space to the user terminal, and the user terminal can display the virtual object in the virtual space according to the layout information.

[0094] As an application example, when the user triggers a virtual object layout instruction on the application of the user terminal, the user can move around the contours of all the physical objects in the physical space for one circle, and the millimeter-wave radar and the camera start to cooperate. The camera can capture an image of all the physical objects in the physical space, and obtain information such as the identification information, the first contour information, and the coordinates of each physical object through semantic segmentation and other methods, and then transmit this information to the millimeter-wave radar. The application of the user terminal can also provide indication information, which can guide the user to move around the contours of all the physical objects in the physical space for one circle, and the millimeter-wave radar can record the corresponding trajectory. After the recording is completed, the millimeter-wave radar can match the first contour information and the trajectory to obtain the layout information of each virtual object in the virtual space. This layout information can be transmitted to the user terminal, and the user terminal can display each virtual object in the virtual space according to the layout information. The user terminal can fill the area in the virtual space enclosed by the trajectory with corresponding stickers, and skip the physical objects that cannot be matched successfully. After the stickers are filled, the user can trigger a layout completion instruction on the application of the user terminal to end the automatic layout function.

[0095] The application of the user terminal can also provide a layout editing function. If the user is not satisfied with the result of the automatic layout, the user can make manual adjustments, which can include adjustments to the size and position. This embodiment can automatically obtain the layout information of the virtual object in the virtual space for the user, thereby improving the layout efficiency and convenience of the virtual object.

[0096] In one embodiment, obtaining the layout information of the virtual object in the virtual space according to the first contour information and the trajectory in step S703 of the above embodiment may include:

[0097] Obtaining second contour information according to the trajectory; if the first contour information matches the second contour information, obtaining the layout position information of the corresponding physical object according to the trajectory, and obtaining the layout information of the virtual object in the virtual space according to the layout position information and the obtained identification information of the physical object.

[0098] In this embodiment, the millimeter-wave radar may use the trajectory as the second contour information, and may obtain the identification information of the physical object. The identification information may be obtained by the camera according to the captured image, and the millimeter-wave radar may receive the identification information of the physical object from the camera. The millimeter-wave radar may match the first contour information with the second contour information. If the first contour information matches the second contour information, the millimeter-wave radar may obtain the layout position information of the physical object according to the trajectory, and obtain the layout information of the virtual object in the virtual space according to the layout position information and the obtained identification information of the physical object.

[0099] In one embodiment, the first contour information may include the first contour information of multiple physical objects, and the second contour information may include the second contour information of multiple physical objects. Based on this, referring to Figure 6 , the method for obtaining the layout information of the virtual object in the present application may also match the first contour information and the second contour information through the following steps, which may include:

[0100] Step S601, determining the first contour information that does not meet the preset contour similarity condition among the first contour information of multiple physical objects.

[0101] Step S602, performing a first match between the first contour information that does not meet the preset contour similarity condition and the second contour information of multiple physical objects.

[0102] Step S603, determining the positional relationship between the unmatched first contour information and the matched first contour information among the first contour information of multiple physical objects.

[0103] Step S604, performing a second match between the unmatched first contour information and the unmatched second contour information among the second contour information of multiple physical objects according to the positional relationship.

[0104] In one embodiment, determining the first contour information that does not meet the preset contour similarity condition among the first contour information of multiple physical objects in step S601 of the method for obtaining the layout information of the virtual object in the present application may also include:

[0105] Cluster the first contour information of multiple entity objects according to contour similarity to obtain multiple first contour information groups of different categories; determine the first contour information that does not meet the preset contour similarity condition in the first contour information of the multiple entity objects according to the number of first contour information included in each of the multiple first contour information groups.

[0106] In one embodiment, the virtual object layout information acquisition method of the present application can be applied to a smart home system as Figure 1 shown. In this embodiment, the virtual object layout information acquisition method is described by taking its application to a smart home system as an example. The smart home system may include a camera, a millimeter-wave radar, a gateway, a user terminal, etc.

[0107] In this embodiment, taking the living room as an example of the entity space, the camera can capture an image of the living room containing various furniture, and obtain the first contour information of various furniture included in the image according to the image. The camera can also obtain the identification information of various furniture included in the image according to the image. The camera can input the captured image of the living room containing various furniture into a trained semantic segmentation model, and obtain information such as the first contour information of various furniture included in the image output by the trained semantic segmentation model, and the identification information of the various furniture. The camera can transmit information such as the first contour information and identification information of the various furniture to the millimeter-wave radar.

[0108] For the above-mentioned trained semantic segmentation model, a U-NET neural network can be used. The image of the living room containing various furniture can be input into the trained semantic segmentation model to obtain the image regions of various furniture in the image output by the semantic segmentation model. The image regions can include all pixels occupied by various furniture in the image, and then the first contour information of each furniture can be extracted according to the image regions. After the image of the living room containing various furniture is input into the trained semantic segmentation model, the semantic segmentation model can also output the identification information of various furniture included in the image. The identification information can be the category information (sofa, table, chair, etc.) to which the furniture belongs.

[0109] For the training process of the above-mentioned trained semantic segmentation model, many images containing furniture such as sofas, tables, chairs, and coffee tables can be collected by the camera of the smart home system as image training samples. Then, the image regions where these furniture are located in the image training samples can be calibrated, and the identification information of the furniture in the image training samples can be calibrated to obtain the calibration information of the image training samples. The calibration information can include the information calibrated for the image regions where the furniture are located in the image training samples and the information calibrated for the identification information of the furniture in the image training samples. For example, the pixels occupied by various furniture can be calibrated in the image training samples, and identification information (1, 2, 3, 4, 5, etc.) can be calibrated for the furniture in the image training samples. Then, the semantic segmentation model to be trained can be trained according to the image training samples and the calibration information. The image training samples can be divided into a training set and a test set according to a ratio such as 7:3. Initial network model parameters can be set for the semantic segmentation model to be trained. After passing each image training sample in the training set through the semantic segmentation model to be trained, the prediction results of the identification information of each pixel can be obtained, and the semantic segmentation model to be trained can be adjusted according to the difference between the prediction results of the identification information of each pixel and the calibration information until the semantic segmentation model meets certain requirements when tested in the test set, such as the pixel segmentation accuracy rate in the test set being greater than the preset accuracy rate threshold, etc., thereby obtaining the trained semantic segmentation model.

[0110] The millimeter-wave radar can obtain information such as the first contour information and identification information of various furniture transmitted by the camera. The client can instruct the user to walk around each piece of furniture in the living room. The millimeter-wave radar detects the trajectory of the user walking around various furniture in the living room. Among them, the user can walk around various furniture in the living room, and the millimeter-wave radar can record its trajectory. After all the furniture in the living room has been walked around by the user in a circle, each piece of furniture can correspond to a trajectory. The millimeter-wave radar can use the trajectory corresponding to each piece of furniture as the second contour information of each piece of furniture. Thus, the millimeter-wave radar can obtain multiple first contour information and multiple second contour information, and then the millimeter-wave radar can perform matching processing on these first contour information and second contour information.

[0111] In the matching process of the millimeter-wave radar for multiple first contour information and multiple second contour information, first, according to the contour similarity, multiple first contour information can be clustered, so that those with the same or similar contours can be grouped into one category. For example, the first contour information is clustered into 5 categories, forming 5 groups of first contour information , and then, according to the number of first contour information included in each of the multiple groups of first contour information, the first contour information that does not meet the preset contour similarity condition among the multiple first contour information can be determined. For example, among the 5 groups of first contour information the group of first contour information that only contains one piece of first contour information The first contour information determined to not meet the preset contour similarity condition. Then, the first contour information group is preferentially subjected to a first match with multiple second contour information, so that furniture with a unique contour can be preferentially determined. If the first contour information group that meets the preset contour similarity condition is matched with multiple second contour information, it will match successfully with multiple second contour information, and in this way, it is impossible to determine which second contour information this first contour information group is matched with. At this time, the first contour information that has been matched and does not meet the preset contour similarity condition can be used and the position relationship (such as relative distance relationship, relative orientation relationship, etc.) between the first contour information group that meets the preset contour similarity condition and has not been matched to determine. For example Figure 5 in, the contour of object 1 is unique among them. The first contour information of object 1 is preferentially matched with each second contour information, and it is very easy to match the second contour information of object 1 corresponding to it. When the first contour information of object 2 is matched with each second contour information, the first contour information of object 2 may match successfully with the second contour information of object 2 and the second contour information of object 3. However, according to the position relationship, the contour of object 2 is closer to the contour of object 1 than the contour of object 3, and the position of object 2 is closer to object 1 than the position of object 3. Based on this, the first contour information of object 2 can be matched with the second contour information of object 2, and thus the first and second contour information of each piece of furniture can be accurately and quickly matched.

[0112] After the contour information is successfully matched, the millimeter-wave radar can determine the corresponding furniture sticker in the furniture sticker library (refer to Figure 3 , which can be provided by the smart home application program of the user terminal) according to the identification information of each piece of furniture, and then obtain the layout information of each piece of furniture in the virtual living room according to the furniture stickers corresponding to each piece of furniture and the layout position information of each piece of furniture.

[0113] After obtaining the layout information of each piece of furniture in the virtual living room, the millimeter-wave radar can send the layout information of each piece of furniture in the virtual living room to the user terminal, and the user terminal can display each furniture sticker in the virtual living room according to this layout information.

[0114] The solution of this application embodiment can automatically complete the acquisition of furniture layout information in the virtual space through the cooperation of the millimeter-wave radar and the camera in the smart home system, reduce the manual operation in the acquisition of layout information, and also avoid the errors caused by manual operation, providing an accurate and reliable data basis for the furniture layout in the virtual space and also providing convenience for the furniture layout in the virtual space.

[0115] In one embodiment, the virtual object layout method of the present application can be applied to a smart home system as shown in Figure 1 The smart home system is described in this embodiment with the virtual object layout method applied to it. The smart home system may include a camera, a millimeter-wave radar, a gateway, a user terminal, and the like.

[0116] In this embodiment, taking the living room as an example of the physical space, the user can trigger a layout instruction such as "automatically generate a home layout" on the smart home application of the user terminal. After the smart home application of the user terminal receives the layout instruction, it can instruct the camera to take an image of the living room containing various pieces of furniture. The camera can obtain information such as the identification information, the first contour information, and the coordinates of each piece of furniture through semantic segmentation and other means, and then these information can be transmitted by the camera to the millimeter-wave radar. The application of the user terminal can also provide indication information to guide the user to move around the contours of all the furniture in the living room for one circle. The application of the user terminal also instructs the millimeter-wave radar to detect and record the trajectory of the user's movement around the contours of the furniture. The millimeter-wave radar can use the trajectory corresponding to the furniture as the second contour information of the furniture. After the recording is completed, the millimeter-wave radar can match these first and second contour information to obtain the layout information of each piece of furniture in the virtual living room (the matching process of the millimeter-wave radar for these first and second contour information can refer to the relevant content in the above application embodiments of the virtual object layout information acquisition method). The millimeter-wave radar can transmit the layout information of each piece of furniture in the virtual living room to the user terminal. The user terminal displays the furniture stickers of each piece of furniture in the virtual living room according to the layout information. The user terminal can fill the area in the virtual living room enclosed by the trajectory with the corresponding furniture stickers. For the furniture that fails to match, it can be skipped. After the furniture stickers are filled, the user can trigger a layout completion instruction on the smart home application of the user terminal to end the automatic layout function. The smart home application of the user terminal can also provide a layout editing function. If the user is not satisfied with the result of the automatic layout, manual adjustment can be made, which can include adjustment of size and position.

[0117] The solution of this application embodiment provides convenience for the furniture layout in the virtual space and eliminates the manual configuration operation. After the user triggers the layout instruction in the smart home application, and moves around each piece of furniture for one circle, the acquisition of the furniture layout information in the virtual space can be completed through the cooperation of the camera and the millimeter-wave radar, and accordingly, the furniture layout in the virtual space corresponding to the physical space is automatically displayed for the user on the interface of the smart home application, without the need for the user to perform cumbersome operations such as selecting and dragging furniture stickers for each piece of furniture, improving the efficiency and convenience of the furniture layout in the virtual space.

[0118] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown in the direction of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless specifically stated herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0119] Based on the same inventive concept, the present application also provides a virtual object layout device for implementing the virtual object layout method described above, and a virtual object layout information acquisition device for the virtual object layout information acquisition method. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more of the following device embodiments can refer to the limitations on the related method in the above text, and will not be repeated here.

[0120] In one embodiment, as Figure 8 shown, a virtual object layout device is provided. The device 800 may include:

[0121] An instruction receiving module 801, configured to receive a virtual object layout instruction;

[0122] A layout acquisition module 802, configured to obtain layout information of a virtual object in a virtual space in response to the virtual object layout instruction; the layout information is obtained according to an image of an entity space including an entity object and a trajectory around the entity object; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object;

[0123] An object display module 803, configured to display the virtual object in the virtual space according to the layout information.

[0124] In one embodiment, the layout acquisition module 802 is configured to obtain first contour information and obtain second contour information; the first contour information is obtained according to the image; the second contour information is obtained according to the trajectory; if the first contour information matches the second contour information, then according to the identification information and layout position information of the entity object, obtain the layout information of the virtual object in the virtual space; the identification information is obtained according to the image; the layout position information is obtained according to the trajectory.

[0125] In one embodiment, a layout acquisition module 802 is configured to determine a corresponding virtual object in a virtual object library according to the identification information; and obtain layout information of the virtual object in a virtual space according to the virtual object and layout position information.

[0126] In one embodiment, the first contour information includes first contour information of a plurality of entity objects; the second contour information includes second contour information of the plurality of entity objects; the layout acquisition module 802 is further configured to determine first contour information that does not meet a preset contour similarity condition among the first contour information of the plurality of entity objects; perform a first match between the first contour information that does not meet the preset contour similarity condition and the second contour information of the plurality of entity objects; determine a positional relationship between the unmatched first contour information and the matched first contour information among the first contour information of the plurality of entity objects; and perform a second match between the unmatched first contour information and the unmatched second contour information among the second contour information of the plurality of entity objects according to the positional relationship.

[0127] In one embodiment, the layout acquisition module 802 is further configured to cluster the first contour information of the plurality of entity objects according to contour similarity to obtain a plurality of first contour information groups of different categories; and determine first contour information that does not meet the preset contour similarity condition among the first contour information of the plurality of entity objects according to the number of first contour information included in each of the plurality of first contour information groups.

[0128] In one embodiment, the layout acquisition module 802 is further configured to input the image into a trained semantic segmentation model; obtain an image region of the entity object in the image output by the semantic segmentation model; obtain the first contour information according to the image region; and obtain identification information of the entity object included in the image output by the semantic segmentation model.

[0129] In one embodiment, the layout acquisition module 802 is further configured to obtain image training samples of various types of entity objects and obtain calibration information of the image training samples; the calibration information includes information for calibrating an image region where an entity object is located in the image training sample and information for calibrating identification information of the entity object in the image training sample; and train a semantic segmentation model to be trained according to the image training samples and the calibration information.

[0130] In one embodiment, as Figure 9 shown, a virtual object layout information acquisition device is provided, and the device 900 may include:

[0131] An information acquisition module 901, configured to acquire first contour information; the first contour information is obtained according to an image of an entity space including an entity object;

[0132] A trajectory acquisition module 902, configured to acquire a trajectory around the entity object;

[0133] A layout obtaining module 903, configured to obtain layout information of a virtual object in a virtual space according to the first contour information and the trajectory; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object.

[0134] In one embodiment, the layout obtaining module 903 is configured to obtain second contour information according to the trajectory; if the first contour information matches the second contour information, obtain layout position information of the entity object according to the trajectory, and obtain the layout information of the virtual object in the virtual space according to the layout position information and the obtained identification information of the entity object; the identification information is obtained according to the image.

[0135] In one embodiment, the first contour information includes first contour information of multiple entity objects; the second contour information includes second contour information of multiple entity objects; the layout obtaining module 903 is further configured to determine first contour information that does not meet a preset contour similarity condition among the first contour information of the multiple entity objects; perform a first match between the first contour information that does not meet the preset contour similarity condition and the second contour information of the multiple entity objects; determine the positional relationship between the unmatched first contour information and the matched first contour information among the first contour information of the multiple entity objects; and perform a second match between the unmatched first contour information and the unmatched second contour information among the second contour information of the multiple entity objects according to the positional relationship.

[0136] In one embodiment, the layout obtaining module 903 is further configured to cluster the first contour information of the multiple entity objects according to contour similarity to obtain multiple first contour information groups of different categories; and determine first contour information that does not meet the preset contour similarity condition among the first contour information of the multiple entity objects according to the number of first contour information included in each of the multiple first contour information groups.

[0137] Each module in the above device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the electronic device in hardware form or be independent of it, or be stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the above respective modules.

[0138] In one embodiment, an electronic device is provided. The electronic device can be a user terminal, a millimeter-wave radar, etc. The internal structure diagram thereof can be as shown in Figure 10 . The electronic device includes a processor, a memory, an input / output interface, and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface is used to exchange information between the processor and external devices. The communication interface is used to communicate with external devices in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a virtual object layout method and a virtual object layout information acquisition method.

[0139] Those skilled in the art can understand that Figure 10 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0140] In one embodiment, an electronic device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0142] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0144] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0145] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0146] The above-described embodiments only represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for virtual object layout, characterized in that, The method includes: Receiving a virtual object layout instruction; In response to the virtual object layout instruction, obtaining layout information of the virtual object in the virtual space; the layout information is obtained based on an image of the physical space including physical objects and a trajectory around the physical objects; the virtual space corresponds to the physical space; the virtual object corresponds to the physical object; Displaying the virtual object in the virtual space according to the layout information.

2. The method according to claim 1, wherein The obtaining the layout information of the virtual object in the virtual space includes: Obtaining first contour information and obtaining second contour information; the first contour information is obtained based on the image; the second contour information is obtained based on the trajectory; If the first contour information matches the second contour information, obtaining the layout information of the virtual object in the virtual space according to the identification information and layout position information of the physical object; the identification information is obtained based on the image; the layout position information is obtained based on the trajectory.

3. The method according to claim 2, wherein The obtaining the layout information of the virtual object in the virtual space according to the identification information and layout position information of the physical object includes: Determining the corresponding virtual object in the virtual object library according to the identification information; Obtaining the layout information of the virtual object in the virtual space according to the virtual object and the layout position information.

4. The method according to claim 2, wherein The first contour information includes the first contour information of multiple physical objects; the second contour information includes the second contour information of multiple physical objects; the method further includes: Determining the first contour information among the first contour information of the multiple physical objects that does not meet the preset contour similarity condition; Performing a first match between the first contour information that does not meet the preset contour similarity condition and the second contour information of the multiple physical objects; Determining the positional relationship between the unmatched first contour information among the first contour information of the multiple physical objects and the matched first contour information among the first contour information of the multiple physical objects; Performing a second match between the unmatched first contour information among the first contour information of the multiple physical objects and the unmatched second contour information among the second contour information of the multiple physical objects according to the positional relationship.

5. The method according to claim 4, characterized in that, The determining the first contour information among the first contour information of the multiple physical objects that does not meet the preset contour similarity condition includes: Clustering the first contour information of the multiple physical objects according to contour similarity to obtain multiple first contour information groups of different categories; Determining the first contour information among the first contour information of the multiple physical objects that does not meet the preset contour similarity condition according to the number of first contour information included in each of the multiple first contour information groups.

6. The method according to claim 2, wherein The obtaining the first contour information includes: Inputting the image into a trained semantic segmentation model; Obtaining the image region of the physical object in the image output by the semantic segmentation model; Obtaining the first contour information according to the image region; After inputting the image into the trained semantic segmentation model, the method further includes: Obtain the identification information of the entity objects included in the image output by the semantic segmentation model.

7. The method according to claim 6, wherein The method further includes: Obtain the image training samples of various entity objects, and obtain the calibration information of the image training samples; the calibration information includes the information for calibrating the image area where the entity objects are located in the image training samples, and the information for calibrating the identification information of the entity objects in the image training samples; Train the semantic segmentation model to be trained according to the image training samples and the calibration information.

8. A method for obtaining virtual object layout information, characterized in that, The method includes: Obtain the first contour information; the first contour information is obtained from the image of the entity space containing the entity objects; Obtain the trajectory around the entity object; According to the first contour information and the trajectory, obtain the layout information of the virtual object in the virtual space; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object.

9. The method according to claim 8, characterized in that, The obtaining the layout information of the virtual object in the virtual space according to the first contour information and the trajectory includes: Obtain the second contour information according to the trajectory; If the first contour information matches the second contour information, obtain the layout position information of the entity object according to the trajectory, and according to the layout position information and the obtained identification information of the entity object, obtain the layout information of the virtual object in the virtual space; the identification information is obtained from the image.

10. The method according to claim 9, wherein The first contour information includes the first contour information of multiple entity objects; the second contour information includes the second contour information of multiple entity objects; the method further includes: Determine the first contour information among the first contour information of the multiple entity objects that does not meet the preset contour similarity condition; Perform a first match between the first contour information that does not meet the preset contour similarity condition and the second contour information of the multiple entity objects; Determine the positional relationship between the unmatched first contour information among the first contour information of the multiple entity objects and the matched first contour information among the first contour information of the multiple entity objects; According to the positional relationship, perform a second match between the unmatched first contour information among the first contour information of the multiple entity objects and the unmatched second contour information among the second contour information of the multiple entity objects.

11. The method according to claim 10, wherein The determining the first contour information among the first contour information of the multiple entity objects that does not meet the preset contour similarity condition includes: Cluster the first contour information of the multiple entity objects according to contour similarity to obtain multiple first contour information groups of different categories; Determine the first contour information among the first contour information of the multiple entity objects that does not meet the preset contour similarity condition according to the number of first contour information included in each of the multiple first contour information groups.

12. A virtual object layout device, characterized in that, The device includes: An instruction receiving module, configured to receive a virtual object layout instruction; A layout obtaining module, configured to obtain the layout information of the virtual object in the virtual space in response to the virtual object layout instruction; the layout information is obtained from the image of the entity space containing the entity objects and the trajectory around the entity object; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object; An object display module, configured to display the virtual object in the virtual space according to the layout information.

13. An apparatus for obtaining virtual object layout information, characterized in that The device includes: An information acquisition module, configured to acquire first contour information; the first contour information is obtained according to an image of an entity space including an entity object; A trajectory acquisition module, configured to acquire a trajectory around the entity object; A layout obtaining module, configured to obtain layout information of a virtual object in a virtual space according to the first contour information and the trajectory; the virtual space corresponds to the entity space; the virtual object corresponds to the entity object.

14. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7 or any one of claims 8 to 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7 or any one of claims 8 to 11.

Images