Data processing apparatus, data processing method, and data processing program

Abstract

To provide a data processor, a data processing method, and a program capable of efficiently generating three dimensional model data of a virtual facility.SOLUTION: The data processing device includes a processing unit that generates first three dimensional model data by inputting first sensor data obtained by measuring an inside of a target facility with a sensor and an instruction to generate a three dimensional model according to the first sensor data to a generation model, and an output unit that outputs a virtual facility corresponding to the target facility using the first three dimensional model data generated by the processing unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] Patent Document 1 discloses a method for constructing a virtual store in a virtual space using three-dimensional CG (Computer Graphics). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-283022 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, virtual facilities such as virtual stores have been realized in virtual spaces. It is desirable to be able to efficiently generate 3D model data for such virtual facilities. However, the technology described in Patent Document 1 leaves room for improvement in terms of efficiently generating 3D model data for virtual facilities. [Means for solving the problem]

[0005] A first aspect of the technology of the present disclosure is a data processing device that includes: a processing unit that generates first three-dimensional model data by inputting first sensor data obtained by measuring the interior of a target facility with a sensor and instructions for generating a three-dimensional model corresponding to the first sensor data into a generative model; and an output unit that outputs a virtual facility corresponding to the target facility using the first three-dimensional model data generated by the processing unit.

[0006] A second aspect of the technology of the present disclosure is a data processing method in which a computer executes a process including generating first three-dimensional model data by inputting first sensor data obtained by measuring the interior of a target facility with a sensor and instructions for generating a three-dimensional model corresponding to the first sensor data into a generative model, and outputting a virtual facility corresponding to the target facility using the generated first three-dimensional model data.

[0007] A third aspect of the technology of the present disclosure is a program for causing a computer to execute a process including generating first three-dimensional model data by inputting first sensor data obtained by measuring the interior of a target facility with a sensor and instructions for generating a three-dimensional model corresponding to the first sensor data into a generative model, and outputting a virtual facility corresponding to the target facility using the generated first three-dimensional model data. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Figure 2] FIG. 2 is a conceptual diagram illustrating an example of main functions of a data processing device. [Figure 3] An example of design drawing data is shown below. [Figure 4] 2 shows a schematic functional configuration of a specific processing unit of the data processing device. [Figure 5] 10 shows an example of second three-dimensional model data. [Figure 6] 1 shows an example of first three-dimensional model data. [Figure 7] An example of a mark displayed on a map is shown below. [Figure 8] 10 is a diagram illustrating an example of an operational flow of specific processing by a data processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] FIG. 1 shows an example of the configuration of a data processing system 10 according to an embodiment. As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server. An example of the smart device 14 is a smartphone. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure.

[0017] The data processing device 12 includes a computer 22 and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The communication I / F 26 is also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the communication I / F 44 are also connected to the bus 52.

[0019] The reception device 38 is equipped with a touch panel, a microphone, etc., and receives user input. The touch panel detects contact with an indicator (e.g., a pen or a finger) to receive user input by the indicator. The microphone detects the user's voice to receive user input by voice. The processor 46 transmits data indicating the user input received by the reception device 38 to the data processing device 12. In the data processing device 12, a specific processing unit 290, which will be described later, acquires the data indicating the user input.

[0020] The output device 40 includes a display and a speaker, and presents data to the user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). The display displays visual information such as text and images in accordance with instructions from the processor 46. The speaker outputs audio in accordance with instructions from the processor 46.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] The data processing device 12 according to this embodiment has a function of constructing a virtual facility corresponding to a facility that actually exists in real space. Hereinafter, a facility for which the data processing device 12 constructs a virtual facility will be referred to as a "target facility." In this embodiment, an example will be described in which a store that sells products is used as the target facility. Note that the target facility is not limited to a store, and may also be a facility where an exhibition is held, etc.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12. As shown in FIG. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "data processing program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the data processing program executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specification processing unit 290. The data generation model 58 is an example of a "generative model" according to the technology of the present disclosure.

[0025] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as image data, voice data, and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0026] The storage 32 also stores design drawing data 70. As shown in Fig. 3, the design drawing data 70 includes dimensions and the like that represent the structure of the building of the target facility, such as a structural drawing of the target facility. The design drawing data 70 may also include the positions and dimensions and the like of equipment that has been pre-installed in the target facility. Examples of pre-installed equipment include built-in lighting and shelves.

[0027] The storage 32 also stores sensor data 72. The sensor data 72 includes measurement results obtained by measuring the interior of the target facility using a sensor. The sensor data 72 is an example of "first sensor data" according to the technology of the present disclosure. Examples of sensors include a LiDAR (Light Detection and Ranging or Laser Imaging Detection and Ranging) and an image sensor provided in a smartphone. The sensor data 72 according to this embodiment includes point cloud data to which color information has been added. Such point cloud data is created, for example, by adding color information obtained by photographing the interior of the target facility using an image sensor to point cloud data obtained by measuring the distance to the interior of the target facility using a LiDAR.

[0028] Furthermore, the storage 32 stores inventory data 74. The inventory data 74 includes the inventory of products sold at the target facility. The inventory data 74 is updated by linking with an inventory management system of the target facility, such as a POS (Point Of Sale) system.

[0029] The storage 32 also stores sensor data 76. The sensor data 76 includes measurement results obtained by measuring products displayed in the facility using a sensor. The sensor data 76 is an example of "second sensor data" according to the technology of the present disclosure. The sensor data 76 includes point cloud data to which color information has been added, similar to the sensor data 72. Such point cloud data is created, for example, by adding color information obtained by photographing the products using an image sensor to point cloud data obtained by measuring the products using a LiDAR.

[0030] Next, the processing of the specific processing unit 290 when the data processing device 12 generates three-dimensional model data and performs specific processing to output the generated three-dimensional model data will be described.

[0031] As shown in FIG. 4, the specific processing unit 290 includes a processing unit 294, an input unit 295, and an output unit 296.

[0032] The processing unit 294 performs a specification process using the data generation model 58. Specifically, the processing unit 294 generates 3D model data by inputting design drawing data 70 and instructions for generating a 3D model corresponding to the design drawing data 70 into the data generation model 58. Hereinafter, the 3D model data generated based on the design drawing data 70 will be referred to as "second 3D model data" to distinguish it from the first 3D model data described below. An example of an instruction for generating a 3D model corresponding to the design drawing data 70 is an instruction such as "Please generate a 3D model in AAA format from the attached design drawing." An example of the data format of the 3D model data is the OBJ format. As a result, 3D model data reflecting the building structure of the target facility is generated, as shown in FIG. 5 as an example.

[0033] Next, the processing unit 294 generates three-dimensional model data (hereinafter referred to as "first three-dimensional model data") by inputting the sensor data 72 and an instruction to generate a three-dimensional model based on the sensor data 72 on the second three-dimensional model data into the data generation model 58. An example of an instruction to generate a three-dimensional model based on the sensor data 72 is a sentence instructing to integrate the sensor data 72 into the second three-dimensional model. As a result, three-dimensional model data is generated that reflects color information of the interior of the target facility and equipment installed inside the target facility, as shown in FIG. 6 as an example.

[0034] The processing unit 294 may generate the first three-dimensional model data by inputting the sensor data 72 and an instruction to generate a three-dimensional model according to the sensor data 72 to the data generation model 58. In this case, the design drawing data 70 does not need to be stored in the storage 32.

[0035] Furthermore, the processing unit 294 generates 3D model data (hereinafter referred to as "third 3D model data") by inputting the sensor data 76 and instructions for generating a 3D model based on the sensor data 76 into the data generation model 58. As a result, 3D model data of a product handled at the target facility is generated, for example. An example of an instruction for generating a 3D model based on the sensor data 76 is an instruction such as "Please generate a 3D model in AAA format from the attached point cloud data." The processing unit 294 may upload the third 3D model data to a cloud server or the like. This allows the 3D model data of the product to be shared with other facilities that handle the same product as the target facility.

[0036] The processing unit 294 may generate third three-dimensional model data using as input three-dimensional model data generated based on a product blueprint or the like, in addition to the sensor data 76 and an instruction to generate a three-dimensional model according to the sensor data 76. The product blueprint or the like may specifically be a product blueprint or a blueprint of the product's exterior. The three-dimensional model data generated based on the product blueprint or the like may be generated by inputting the product blueprint or the like and an instruction to generate a three-dimensional model according to the product blueprint or the like into the data generation model 58. Furthermore, the three-dimensional model data generated based on the product blueprint or the like may be data provided from an external source.

[0037] Furthermore, the processing unit 294 sets a mark for accessing a virtual facility corresponding to the target facility at the position of the target facility on the map displayed by the map software. As a result, as shown in Fig. 7 as an example, a mark M for accessing the virtual facility is displayed at the position of the target facility on the map displayed by the map software.

[0038] A user of the data processing system 10 accesses the virtual facility via the reception device 38 of the smart device 14. This access may be made from the above-mentioned mark M, or, for example, from a website that has a link to access the virtual facility.

[0039] The input unit 295 acquires a user input received by the smart device 14. Specifically, the input unit 295 receives access to the virtual facility by the user.

[0040] In response to the user's access to the virtual facility, the output unit 296 outputs a virtual facility corresponding to the target facility to the smart device 14 using the first three-dimensional model data generated by the processing unit 294. At this time, the output unit 296 places a virtual product corresponding to the product in the virtual facility using the third three-dimensional model data generated by the processing unit 294. As a result, the virtual facility in which the virtual product is placed is displayed on the display of the output device 40 of the smart device 14.

[0041] For example, the output unit 296 places virtual products in a virtual facility using a trained model obtained by machine learning. This trained model is, for example, a machine learning model that outputs the placement positions of the virtual products in the virtual facility when three-dimensional model data representing the virtual facility and three-dimensional model data representing the virtual products are input. Note that the output unit 296 may place the virtual products at positions in the virtual facility that are set in advance for each virtual product.

[0042] Furthermore, the output unit 296 refers to the inventory data 74, and when the inventory of a virtual product corresponding to the virtual product placed in the virtual facility has run out in the target facility, the output unit 296 places a virtual product corresponding to a different product in place of the virtual product corresponding to the virtual product. In this case, an example of the substitute virtual product is a virtual product that sells well to customers of the same age and gender as the product whose inventory has run out.

[0043] Next, the operation of the data processing system 10 will be described. An example of the flow of the specific processing will be described with reference to Fig. 8. The flow of the specific processing shown in Fig. 8 is an example of a "data processing method" according to the technology of the present disclosure.

[0044] In step S300, as described above, the processing unit 294 generates second three-dimensional model data by inputting the design drawing data 70 and an instruction to generate a three-dimensional model corresponding to the design drawing data 70 into the data generation model 58. In step S302, as described above, the processing unit 294 generates first three-dimensional model data by inputting the sensor data 72 and an instruction to generate a three-dimensional model corresponding to the sensor data 72 on the second three-dimensional model data into the data generation model 58.

[0045] In step S304, as described above, the processing unit 294 generates third 3D model data by inputting the sensor data 76 and instructions for generating a 3D model based on the sensor data 76 into the data generation model 58. In step S306, the processing unit 294 sets a mark for accessing a virtual facility corresponding to the target facility at the position of the target facility on the map displayed by the map software.

[0046] In step S308, the input unit 295 waits until it acquires user input via the reception device 38 of the smart device 14. When the user performs an input operation to access the virtual facility via the reception device 38 of the smart device 14, the determination in step S308 becomes positive, and the process proceeds to step S310.

[0047] In step S310, as a response to the access to the virtual facility in step S308, the output unit 296 outputs a virtual facility corresponding to the target facility to the smart device 14 using the first three-dimensional model data generated in step S302. In step S312, the output unit 296 places a virtual product corresponding to the product in the virtual facility using the third three-dimensional model data generated in step S304.

[0048] In step S314, the output unit 296 refers to the inventory data 74 and determines whether any of the virtual products corresponding to the virtual products placed in the virtual facility has become out of stock in the target facility. If this determination is positive, the process proceeds to step S316.

[0049] In step S316, the output unit 296 places a virtual product corresponding to a different product in the virtual facility in place of the virtual product corresponding to the product whose stock has run out. When the processing of step S316 ends, the identification process ends. On the other hand, if the determination in step S314 is negative, step S316 is not executed and the identification process ends.

[0050] As described above, according to this embodiment, it is possible to efficiently generate three-dimensional model data of virtual facilities that correspond to facilities that actually exist in real space.

[0051] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[0052] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0053] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0054] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0055] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0056] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0057] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0058] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0059] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0060] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0061] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0062] 10 Data Processing System 12 Data Processing Device 14 Smart Devices 56 Specific Processing Program 290 Special Processing Department 294 Processing Section 295 Input section 296 Output Section< / url:>

Claims

1. a processing unit that generates first three-dimensional model data by inputting first sensor data obtained by measuring the interior of the target facility with a sensor and an instruction to generate a three-dimensional model corresponding to the first sensor data into a generative model; an output unit that outputs a virtual facility corresponding to the target facility using the first three-dimensional model data generated by the processing unit; 2. A data processing device comprising:

2. the processing unit generates second three-dimensional model data by inputting design drawing data of the target facility and an instruction to generate a three-dimensional model according to the design drawing data into the generation model; The first three-dimensional model data is generated by inputting the first sensor data and an instruction to generate a three-dimensional model based on the first sensor data on the second three-dimensional model data into the generative model.

2. The data processing device according to claim 1.

3. the processing unit generates third three-dimensional model data by inputting second sensor data obtained by measuring products displayed in the target facility with the sensor and instructions for generating a three-dimensional model according to the second sensor data into a generative model; The output unit uses the third three-dimensional model data generated by the processing unit to place a virtual product corresponding to the product within the virtual facility.

3. The data processing device according to claim 1.

4. When the inventory of a product becomes zero in the target facility, the output unit places a virtual product corresponding to a product different from the virtual product in place of the virtual product corresponding to the product.

4. The data processing device according to claim 3.

5. The processing unit sets a mark for accessing the virtual facility at the position of the target facility on a map displayed by map software.

3. The data processing device according to claim 1.

6. generating first three-dimensional model data by inputting first sensor data obtained by measuring the interior of the target facility with a sensor and instructions for generating a three-dimensional model according to the first sensor data into a generative model; Using the generated first three-dimensional model data, a virtual facility corresponding to the target facility is output. A data processing method in which a computer executes a process including:

7. generating first three-dimensional model data by inputting first sensor data obtained by measuring the interior of the target facility with a sensor and instructions for generating a three-dimensional model according to the first sensor data into a generative model; Using the generated first three-dimensional model data, a virtual facility corresponding to the target facility is output. A program that causes a computer to execute processes including the above.

Images