Information processing system, information processing method, and information processing program

Abstract

To improve the construction or operation of a facility using a simulator, etc.SOLUTION: In an information processing system, a production facility, an information processing device, and a plurality of user terminals communicate with each other via a communication line. An acquisition unit of a processor of the information processing device acquires design information indicative of a facility layout design and operation information on the operation status of the facility in a real space mounted according to the layout design. A generation unit generates a virtual facility on the basis of the design information and operation information. The virtual facility is a facility reproduced in a prescribed virtual space, and is operated in the virtual space by a learning model. The generation unit also generates at least one learning data on the basis of the results of a prescribed physical simulation to the virtual facility. The learning data is used to learn the learning model of the virtual facility. A learning unit learns the learning model of the virtual facility using the learning data, and outputs the learning parameters of the learning model in the real space on the basis of the learning results.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program.

Background Art

[0002] Patent Document 1 discloses a logistics system that provides a system, apparatus, and method capable of highly accurately detecting and predicting failures of an apparatus using machine learning.

[0003] In the entire process from receiving to shipping at a logistics center, the logistics system obtains identification information of conveyed goods, workers, and handling equipment acquired from sensing means and / or video acquisition means via a communication line, optimizes the flow volume of the conveyed goods and the movement routes of the workers and / or the handling equipment by a 3D simulator on a computer, and displays flow volume information and failure detection and / or failure prediction information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, there is still room for improvement in the construction or operation of facilities using a 3D simulator or the like on such a computer.

Means for Solving the Problems

[0006] According to one aspect of the present invention, an information processing system is provided. This information processing system comprises at least one processor capable of executing a program so as to perform the following steps: In the acquisition step, design information indicating the layout design of equipment and operation information relating to the operational status of the equipment in real space implemented according to the layout design are acquired. The equipment includes a drive device that can be driven based on a learning model that includes at least one learning parameter. In the first generation step, a virtual equipment is generated based on the design information and the operation information. The virtual equipment is equipment reproduced in a predetermined virtual space and is configured to be operational in the virtual space by the learning model. In the second generation step, at least one training data is generated based on the results of a predetermined physical simulation of the virtual equipment. The training data is used to train the learning model of the virtual equipment. In the output step, the learning parameters of the learning model in real space are output based on the results of training the learning model of the virtual equipment using the training data.

[0007] This configuration allows for more efficient construction or operation of the equipment. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the configuration of Information Processing System 1. [Figure 2] This is a block diagram showing the hardware configuration of the information processing device 2. [Figure 3] This is a block diagram showing the hardware configuration of user terminal 3. [Figure 4] This is a diagram showing an example of production equipment 4. [Figure 5] This figure shows an example of the functional components of the processor 23. [Figure 6] This is an activity diagram showing an example of the first information processing flow executed in information processing system 1. [Figure 7] This is a conceptual diagram showing the relationship between production equipment 4 and virtual production equipment 5. [Figure 8]This is an activity diagram showing an example of the flow of the second information processing performed in information processing system 1. [Figure 9] This is an activity diagram showing an example of the processing flow when a stop operation is performed in Information Processing System 1. [Figure 10] This figure shows an example of a workpiece 6. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other.

[0010] Incidentally, the program for implementing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium, or it may be provided so that it can be downloaded from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Furthermore, in this embodiment, "part" may include, for example, hardware resources implemented by circuits in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. In addition, various types of information are handled in this embodiment, and these types of information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on circuits in a broad sense.

[0012] In addition, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit (Circuit), circuitry (Circuitry), a processor (Processor), a memory (Memory), etc. That is, it includes an application specific integrated circuit (ASIC), programmable logic devices (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration will be described.

[0014] <Information Processing System 1> FIG. 1 is a configuration diagram showing an information processing system 1. The information processing system 1 includes an information processing device 2, a user terminal 3, and production equipment 4. The information processing device 2, the user terminal 3, and the production equipment 4 are configured to be communicable through a telecommunication line. In one embodiment, the information processing system 1 is composed of one or more devices or components. For example, if it consists only of the information processing device 2, the information processing system 1 can be the information processing device 2. Hereinafter, these components will be described.

[0015] <Information Processing Device 2> FIG. 2 is a block diagram showing the hardware configuration of the information processing device 2. The information processing device 2 includes a communication unit 21, a storage unit 22, and a processor 23, and these components are electrically connected via a communication bus 20 inside the information processing device 2. Each component will be further described.

[0016] Although the communication unit 21 preferably uses wired communication means such as USB, IEEE 1394, Thunderbolt (registered trademark), and wired LAN network communication, it may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, and BLUETOOTH (registered trademark) communication as required. That is, it is more preferable to implement it as a set of these multiple communication means. That is, the information processing apparatus 2 may communicate various information from the outside via the communication unit 21 and the network.

[0017] The storage unit 22 stores various information defined by the foregoing description. This can be implemented, for example, as a storage device such as a Solid State Drive (SSD) that stores various programs related to the information processing apparatus 2 executed by the processor 23, or as a memory such as a Random Access Memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program operations. The storage unit 22 stores various programs, variables, etc. related to the information processing apparatus 2 executed by the processor 23.

[0018] The processor 23 performs processing and control of the overall operations related to the information processing apparatus 2. The processor 23 is, for example, a Central Processing Unit (CPU) not shown in the figure. The processor 23 realizes various functions related to the information processing apparatus 2 by reading a predetermined program stored in the storage unit 22. That is, the information processing by software stored in the storage unit 22 is specifically realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. These will be described in more detail in the next section. Note that the processor 23 is not limited to being single, and may be implemented to have multiple processors 23 for each function. Or combinations thereof may also be used.

[0019] <User terminal 3> Figure 3 is a block diagram showing the hardware configuration of the user terminal 3. The user terminal 3 comprises a communication unit 31, a storage unit 32, a processor 33, a display unit 34, and an HMI device 35, and these components are electrically connected within the user terminal 3 via a communication bus 30. The descriptions of the communication unit 31, storage unit 32, and processor 33 are the same as the descriptions of each part in the information processing device 2, so they are omitted here.

[0020] The display unit 34 may be included in the user terminal 3 housing or it may be an external component. The display unit 34 displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably done by using different display devices such as a CRT display, liquid crystal display, organic EL display, and plasma display, depending on the type of user terminal 3.

[0021] The HMI device 35 is a human-machine interface device. The HMI device 35 may be included in the housing of the user terminal 3 or it may be an external device. For example, the HMI device 35 may be implemented as a touch panel integrated with the display unit 34. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, instead of a touch panel, a switch button, mouse, QWERTY keyboard, voice recognition device, gesture detection device, gaze detection device, biosignal detection device, imaging device, etc. may be used. In other words, the HMI device 35 receives operation input made by the user. In response, the HMI device 35 transmits a signal corresponding to the operation input to the processor 33 via the communication bus 30. The processor 33 can perform predetermined controls and calculations as needed. The HMI device 35 can also be said to include an input unit configured to accept input from the user.

[0022] <Production Equipment 4> Figure 4 shows an example of production equipment 4. Production equipment 4 is a shared warehouse for producing and managing a predetermined product, and is one embodiment of the equipment. As shown in Figure 4, in this embodiment, production equipment 4 includes receiving equipment 41, shipping equipment 42, storage equipment 43, lighting equipment 44, and a forklift 45 as a drive device. Production equipment 4 may also further include a manufacturing line for producing the product.

[0023] <Warehouse equipment 41> The receiving facility 41 comprises a production line for producing products and a receiving area for receiving the products produced on the production line. The production line includes various production equipment, such as a raw material introduction machine, a raw material processing machine, and a conveyor for transporting objects processed by the introduction machine or processing machine, and these machines are arranged according to a layout specified by the user or the like. The receiving area is, for example, a receiving platform. Pallets loaded with the produced products are placed on this platform.

[0024] <Dispatch Equipment 42> The outbound equipment 42 is equipment for taking products, etc., placed on the inbound equipment 41 out to the outside, and may be, for example, an outbound platform similar to the inbound equipment 41, or a transport vehicle such as a truck.

[0025] <Storage equipment 43> The storage equipment 43 is equipment capable of storing goods such as products, for example, shelves. While the storage equipment 43 functions as equipment for storing incoming products, it also acts as an obstacle when the forklift 45, described later, moves inside the production equipment 4, and defines an area where the forklift 45 is prohibited from entering.

[0026] <Lighting equipment 44> The lighting equipment 44 is capable of adjusting the brightness inside the production equipment 4 by irradiating or introducing light into the production equipment 4, and is, for example, a daylight or a light fixture. When the lighting equipment 44 irradiates or introduces light into the production equipment 4, the visibility inside the production equipment 4 changes. This is because the light is directly irradiated onto the products etc. contained in the production equipment 4, and is scattered by other components contained in the production equipment 4, such as the walls, floor, storage equipment 43 etc. of the production equipment 4, and these components function as indirect lighting for the products etc. Therefore, it functions as a disturbance to the visual information inside the production equipment 4.

[0027] <Forklift 45> The forklift 45 is configured to be drivable based on a learning model M1 which includes at least one learning parameter. The forklift 45 in this embodiment is configured to perform operations such as driving along a route R within the production facility 4 and loading and unloading products according to a predetermined learning model M1. The learning model M1 functions as an identifier configured to distinguish an identification target, such as a pallet loaded with products, from other items. The forklift 45 also stores a map for moving around inside the production facility 4. The format of the map is arbitrary, but for example, it can be represented by an occupancy grid map. In the occupancy grid map, the locations where the receiving facility 41, the shipping facility 42, and the storage facility 43 exist, and the locations where items have been observed, are represented as no-entry zones. The forklift 45 drives along a predetermined route R so as to avoid the occupancy grids corresponding to these no-entry zones. The occupancy grid map may be automatically generated by the forklift 45 or generated by an external device such as the information processing device 2. The forklift 45 of this embodiment comprises a vehicle body 451 capable of traveling inside the production facility 4 in the real world, a loading and unloading device 452 capable of loading and unloading products, and a detection device 453.

[0028] <Detection device 453> The detection device 453 is configured to detect objects around the forklift 45. The detection device 453 in this embodiment includes an imaging camera capable of optically imaging objects around the forklift 45 and is mounted on the vehicle body 451 or the cargo handling device 452. The imaging camera in this embodiment is configured to image the area in front of the forklift 45. The detection device 453 is not limited to an imaging camera, but can be any device capable of detecting information that allows for understanding the positional relationship between the forklift 45 and the objects, such as a LiDAR (Light Detection And Ranging) or ultrasonic detection device. The detection device 453 may also be configured to detect the state of the forklift 45. For example, the detection device 453 may include a speedometer to detect the driving state of the forklift 45 (driving speed, direction, etc.), a detector to detect the state of the cargo handling device 452 (e.g., fork position, tilt angle, etc.).

[0029] 2. Functional configuration of the information processing device 2 Figure 5 shows an example of the functional units of the processor 23. As shown in Figure 5, the processor 23 includes an acquisition unit 231, a generation unit 232, a simulation unit 233, an evaluation unit 234, a learning unit 235, an update unit 236, a presentation unit 237, and a learning stop unit 238. This section will describe the outline of these functional units. Details of each functional unit will be explained later in conjunction with the information processing described below.

[0030] The acquisition unit 231 is configured to acquire information from the user terminal 3 or other devices. The acquisition unit 231 is configured to acquire various information by reading various information stored in the storage area, which is at least a part of the memory unit 22, and writing the read information to the work area, which is at least a part of the memory unit 22. The storage area is, for example, the area of ​​the memory unit 22 that is implemented as a storage device such as an SSD. The work area is, for example, the area that is implemented as memory such as RAM. The acquisition by the acquisition unit 231 includes acquiring the output results of each functional unit included in the processor 23.

[0031] The generation unit 232 is configured to generate a predetermined virtual space and various objects contained within that virtual space based on the acquired information. In this way, the generation unit 232 reproduces the production equipment 4 within the virtual space. The specific manner in which such a virtual space is generated is arbitrary, but it can be realized using various platforms, such as NVIDIA's Omniverse®.

[0032] The simulation unit 233 is configured to perform various physical simulations based on the various information acquired. This allows the simulation unit 233 to reproduce the operation of the production equipment 4, which is recreated in the virtual space, within that virtual space. While the method for performing such various physical simulations in the virtual space is arbitrary, it can be implemented using various simulation tools, such as NVIDIA's Isaac Sim®.

[0033] The evaluation unit 234 is configured to output various evaluation indicators related to a predetermined evaluation target based on the acquired information. The evaluation unit 234 is configured to make judgments regarding various conditions based on the output evaluation indicators.

[0034] The learning unit 235 trains the learning model M1 based on the acquired information, the outputted evaluation indicators, and the results of the physical simulation performed by the simulation unit 233.

[0035] The update unit 236 updates various parameters related to the learning model M1, information related to the layout design of the production equipment 4, and various other information such as the state in the virtual space, based on the acquired information or the learning results from the learning unit 235.

[0036] The display unit 237 is configured to display various types of information. This information can be presented to the user via the display unit 34 of the user terminal 3 or other devices. In such cases, for example, the display unit 237 controls the display unit 34 of the user terminal 3 to display visual information such as screens, images including still images or videos, icons, and messages. The display unit 237 may generate only rendering information for displaying the visual information on the user terminal 3. The display unit 237 may also present the outputted information to the user without going through the user terminal 3 or other devices.

[0037] The learning stop unit 238 is configured to stop learning the learning model M1 when predetermined stopping conditions are met.

[0038] 3. Regarding information processing This section describes the information processing performed in the information processing system 1 mentioned above. The information processing in this embodiment includes a first information processing using physical simulation in a virtual space, a second information processing for recording the relationship between the operation of the forklift 45 and its fitness during the operation of the production equipment 4, and a third information processing performed when a predetermined stopping operation is performed.

[0039] 3.1. Regarding the first information processing flow First, the first information processing flow will be explained. Figure 6 is an activity diagram showing an example of the first information processing flow executed in information processing system 1. Note that this information processing may include any exception handling not shown. Exception handling includes interrupting the information processing or omitting each process. The selection or input performed in this information processing may be based on user operation or may be performed automatically without user operation.

[0040] [Activity A1] First, in Activity A1, the acquisition unit 231 acquires design information IF1, which shows the layout design of the equipment, and operation information IF2. This information is entered by a user, such as the person in charge of designing the production equipment 4.

[0041] Design information IF1 shows the layout design of production equipment 4. Design information IF1 includes, for example, the dimensions of production equipment 4, and the location, size, and scope of receiving equipment 41, shipping equipment 42, storage equipment 43, lighting equipment 44, etc. Design information IF1 may also include information about products or other items to be stored in receiving equipment 41, shipping equipment 42, and storage equipment 43 (e.g., type, size, quantity, storage method, etc.). Design information IF1 is represented by information showing the entire production equipment 4, such as design drawings, architectural drawings, floor plans, 3D CAD data of production equipment 4, and by products or items to be stored inside production equipment 4.

[0042] Operational information IF2 is information regarding the operational status of the production equipment 4 in the real space, which is implemented according to the layout design. Operational information IF2 includes, for example, the location, size, or range of equipment included in the receiving equipment 41, the type of product, the amount brought in, or the amount of items taken out, the operating status or schedule of the lighting equipment 44 (for example, the lighting time and illuminance of the lighting equipment 44), the objects transported by the production equipment 4, or the type of forklift 45, travel time, travel speed, or travel schedule. If the production equipment 4 is already implemented, the acquisition unit 231 acquires the operational information IF2 of the production equipment 4. However, if the production equipment 4 is planned for implementation and has not yet been implemented, the acquisition unit 231 acquires the operational information IF2 of the production equipment 4 that is planned for implementation, i.e., information regarding the planned operation.

[0043] [Activity A2] Next, the process proceeds to activity A2, where the generation unit 232 generates a virtual production facility 5 as a virtual facility based on the design information IF1 and the operation information IF2. The virtual production facility 5 is an object corresponding to the production facility 4 in the virtual space where the physical simulation described later takes place.

[0044] Here, the relationship between production equipment 4 and virtual production equipment 5 will be explained using Figure 7. Figure 7 is a conceptual diagram showing the relationship between production equipment 4 and virtual production equipment 5.

[0045] The virtual production equipment 5, as a virtual facility, is a reproduction of the production equipment 4 in a virtual space and is configured to be operational in the virtual space by the learning model M1. The virtual production equipment 5 includes a virtual receiving facility 51, a virtual shipping facility 52, a virtual storage facility 53, a virtual lighting facility 54, and a virtual forklift 55 as a virtual drive device. The virtual receiving facility 51, virtual shipping facility 52, virtual storage facility 53, virtual lighting facility 54, and virtual forklift 55 are objects in the virtual space that correspond to the receiving facility 41, shipping facility 42, storage facility 43, lighting facility 44, and forklift 45, respectively. These objects may be represented by textures or by a set of coordinate data such as a point cloud. The processor 23 may also store the virtual space thus generated in the memory unit 22 as a virtual occupied grid map that the forklift 45 can grasp.

[0046] The virtual lighting equipment 54 functions as a source of disturbances in the physical simulation performed in the virtual space. For example, the virtual lighting equipment 54 is configured to allow settings such as whether or not to irradiate or introduce light into the virtual production equipment 5, the properties of the irradiated or introduced light, such as intensity, spectrum, and propagation mode (point light source propagation, area light source propagation, etc.). The generation unit 232 can generate disturbance information IF3 related to the light source by configuring the virtual lighting equipment 54.

[0047] The virtual forklift 55 is an object configured to move within the virtual production facility 5. The virtual forklift 55 comprises a virtual body 551, a virtual cargo handling device 552, and a virtual detection device 553, corresponding to the forklift 45. The virtual body 551, virtual cargo handling device 552, and virtual detection device 553 are objects corresponding to the body 451, cargo handling device 452, and detection device 453 in the virtual space, respectively. The virtual forklift 55 can move within the virtual production facility 5 and use the virtual cargo handling device 552 to handle products in the virtual space, such as pallets on which products are loaded. The virtual detection device 553 is configured to detect the state in the virtual space and simulates the sensors included in the virtual production facility 5. In this embodiment, the virtual detection device 553 includes a virtual imaging camera that functions as an imaging camera in the virtual space. The virtual imaging camera is configured to reproduce a scene similar to that of an imaging camera in the real space and can observe the view of the production facility 4 from the perspective of the virtual forklift 55. Therefore, the virtual imaging camera can be described as a simulator that reproduces the observation results from the forklift 45's imaging camera in a virtual space. The observation results from such a virtual detection device 553 can be output, for example, as at least one image data.

[0048] [Activity A3] As shown in Figure 6, after activity A2, processing proceeds to activity A3, where processor 23 sets waypoint P1 (see Figure 7). Waypoint P1 is information about a specific location that the virtual forklift 55 must be included in the path R from the starting position to the target position. Setting waypoint P1 is not limited to simply setting the location of waypoint P1, but may also include setting the actions to be performed by the virtual forklift 55 at waypoint P1 (e.g., loading / unloading, turning, waiting, etc.).

[0049] [Activity A4] Next, the process proceeds to activity A4, where the generation unit 232 generates a route R on which the virtual forklift 55 will travel, based on the generated virtual production equipment 5 and the set waypoint P1. Route R is configured to match the route on which the forklift 45 will travel in the occupied grid map of real space. The generation unit 232 generates a virtual occupied grid map in which objects included in the virtual production equipment 5 are represented as restricted areas, and generates a route R that moves from the virtual receiving equipment 51 to the virtual shipping equipment 52, including waypoint P1. Route R is information that defines the operation of the forklift 45 included in the production equipment 4, and is one aspect of the operation information IF2.

[0050] [Activity A5] Next, the process proceeds to activity A5, and the simulation unit 233 executes a predetermined physical simulation on the virtual production equipment 5. The physical simulation in this embodiment includes an optical simulation. The optical simulation is a simulation using a photorealistic physical simulator, such as a simulation that takes indirect lighting effects into account by ray tracing. This allows the simulation unit 233 to reproduce the appearance of the virtual production equipment 5 with the same light irradiation or scattering as the production equipment 4.

[0051] In this embodiment, the simulation unit 233 causes the photorealistically reproduced virtual production facility 5 to perform the same operations as the production facility 4 implemented in the real world, based on the acquired operational information IF2. By faithfully reproducing the production facility 4 using the virtual production facility 5, the simulation unit 233 can predict in advance the impact of changes or updates to the layout design of the production facility 4.

[0052] [Activity A6] Next, the process proceeds to activity A6, where the evaluation unit 234 outputs an evaluation index indicating the validity of the design information IF1 based on the physical simulation. The evaluation index also indicates the physical consistency of the design information IF1, and its value changes depending on whether, for example, objects within the virtual production facility 5 (such as the virtual receiving facility 51) generated by the design information IF1 interfere with other objects, or whether there is a path R that allows the virtual forklift 55 to move from the virtual receiving facility 51 to the virtual shipping facility 52 due to the arrangement of the virtual storage facility 53, etc. If the evaluation index satisfies the predetermined stop conditions, the process returns to activity A1, the user inputs the design information IF1 etc. into the information processing device 2 again, and the acquisition unit 231 acquires the design information IF1 etc. Therefore, the processor 23 does not perform learning of the learning model M1 if the evaluation index satisfies the stop conditions. Note that the evaluation index is not limited to numerical information, etc., and may be expressed as the conditions described above. This ensures that the learning model is not trained if the learning premise is not valid due to circumstances such as the layout of the equipment being physically difficult to implement. This reduces the waste of computing resources.

[0053] In this embodiment, the display unit 237 further displays stop information relating to the satisfied stop conditions in a manner that is visible to the user when the stop conditions are met. This allows the user to efficiently review the design information based on the stop information. The stop information includes, for example, the content of the stop conditions and details of the output evaluation indicators (for example, information such as route R does not exist or the durability of the storage facility 43 is insufficient).

[0054] [Activity A7] On the other hand, if the stopping conditions are not met, the process proceeds to activity A7, where the travel time of the forklift 45 is calculated based on the results of the physical simulation and presented to the design engineer as the user. Based on the presented results, the design engineer specifies whether or not to change the layout of the production equipment 4. If the design engineer specifies a layout change, the process returns to activity A1, and the acquisition unit 231 acquires new design information IF1 and operation information IF2.

[0055] [Activity A8] Next, the process proceeds to activity A8, and the acquisition unit 231 refers to the storage unit 22, etc., and acquires the low fitness record. The low fitness record is a record of the operation of the production equipment 4 in situations where fitness is low, as described later, and is associated with the operation of the forklift 45, etc., when there is a discrepancy in the operation of the production equipment 4 and the virtual production equipment 5. This low fitness record is used as part of the training data D1 (for example, as a label for the generated training data D1) when training the learning model M1, as described later.

[0056] [Activity A9] Next, the process proceeds to activity A9, where the processor 23 determines the layout of the virtual production equipment 5 in the virtual space based on the design information IF1 and the operation information IF2.

[0057] [Activity A10] Next, the process proceeds to activity A10, where the processor 23 sets the disturbances to be reproduced within the virtual production facility 5 based on the design information IF1 and the operation information IF2. As a result, the generation unit 232 generates disturbance information IF3. Setting disturbances to act on the virtual production facility 5 is one way of obtaining disturbance information IF3 related to disturbances in the production facility 4. For example, the processor 23 sets the upper limit, lower limit, and step width of the illuminance of each lighting equipment 44 within the range defined by the design information IF1 and the operation information IF2, and generates disturbance information IF3 by sampling the illuminance values ​​at intervals of the step width within the defined domain defined by the upper and lower limits.

[0058] [Pier A11] Next, the process proceeds to activity A11, where the processor 23 determines the observation method for the virtual production equipment 5. The processor 23 sets, for example, the position, orientation, field of view, and observation time of the virtual detection device 553, and the operating conditions of the virtual lighting equipment 54. The position of the virtual detection device 553 is one of the observation positions for observing the virtual production equipment 5. The processor 23 may also use observation means different from the virtual detection device 553 to observe the virtual production equipment 5 from viewpoints other than that of the virtual forklift 55.

[0059] [Activity A12] Next, the process proceeds to activity A12, where the simulation unit 233 performs a physical simulation according to the determined observation pattern. This allows the processor 23 to observe the virtual production equipment 5 while updating the observation position of the virtual detection device 553 as the virtual forklift 55 moves within the virtual production equipment 5. The generation unit 232 generates the results of this observation as at least one image data. This image data is used as training data D1 during the training of the learning model M1, which will be described later. In other words, the generation unit 232 generates at least one training data D1 based on the results of a predetermined physical simulation of the virtual production equipment 5. The training data D1 is used to train the learning model M1 of the virtual production equipment 5. Note that the virtual detection device 553 may be a LiDAR or the like, similar to the detection device 453, so the training data D1 includes at least one of the image data and LiDAR data of the virtual production equipment 5.

[0060] In this embodiment, the generation unit 232 further generates disturbance learning data D2 by applying a disturbance to at least one of the learning data D1 based on the disturbance information IF3 in the virtual space. The disturbance learning data D2 is used to train the learning model M1, just like the learning data D1. Therefore, the number of data used to train the learning model M1 can be increased, thereby improving the accuracy of the learning model M1. Furthermore, the disturbance learning data D2 can be said to be learning data D1 that reflects the effects of disturbances. Therefore, by training the learning model M1 using the disturbance learning data D2, changes in the situation in the real space, such as deterioration of lighting, introduction of new equipment, and weather, can be incorporated as disturbances in the physical simulation, and learning parameters that are more applicable to a wider range of situations can be obtained. Consequently, the operational efficiency of the equipment can be improved. The set of training data used to train the learning model M1, such as training data D1 and disturbance training data D2, is called a dataset and is stored in the memory unit 22, etc.

[0061] [Pier A13] Next, the process proceeds to activity A13, where the learning unit 235 trains a learning model M1 of the virtual production equipment 5 based on the dataset. That is, the learning unit 235 trains a learning model M1 of the virtual production equipment 5 based on at least the training data D1. As a result, the processor 23 outputs the learning parameters of the learning model M1 in the real world based on the results of training the learning model M1 of the virtual equipment using the training data D1. In this embodiment, the processor 23 further outputs the learning parameters of the learning model M1 in the real world based on the results of training the learning model M1 of the virtual equipment using the disturbance training data D2. The processor 23 outputs the learning results, such as the learning parameters, in a manner applicable to the learning model M1 of the production equipment 4 in the real world.

[0062] The training data D1 and disturbance training data D2 in this embodiment are images of the virtual production equipment 5 taken from multiple viewpoints, and multiple objects corresponding to pallets, which are the target of identification by the training model M1, are placed on them. Disturbance training data D2 is image data processed from the image data included in training data D1 so that rectangular prism-shaped objects are placed on it based on disturbance information IF3. Rectangular prism-shaped objects are movable items, such as temporarily placed luggage. Such items may interfere with the recognition of pallets based on the detection results of the detection device 453 by the training model M1. Therefore, by pre-generating image data that interferes with such recognition targets based on disturbance information IF3, and training the training model M1 based on this image data, the performance of the training model M1 installed on the actual production equipment 4 can be improved through physical simulation in a virtual space.

[0063] [Activity A14] As shown in Figure 7, the process then proceeds to activity A14, where the processor 23 calculates fitness based on the operational information IF2. Fitness indicates the consistency between the operational status of the virtual equipment and the operational status of the equipment in the real world. The specific form of fitness is arbitrary, but for example, fitness can be defined as the similarity between the occupied grid map in the real world, constructed by sensors such as the detection device 453 included in the production equipment 4, and the virtual occupied grid map in the virtual space, constructed by the virtual detection device 553 included in the virtual production equipment 5. Fitness increases as the consistency (e.g., degree of agreement) between the occupied grid map in production equipment 4 and the virtual occupied grid map in the virtual space increases. Thus, fitness is arbitrary as long as production equipment 4 and virtual production equipment 5 are comparable. Subsequently, the processor 23 determines whether the fitness satisfies predetermined update conditions. The update conditions are conditions for determining whether to update the learning model M1 based on the learning results described above, and in this embodiment, this includes the fitness being less than a threshold. The threshold can be arbitrarily set by the user or others according to the tolerance for the operation of production equipment 4. If the fitness does not meet the update conditions (in this embodiment, if the fitness is above the threshold), the processing of activities A15 to A17 described later is omitted, and the first information processing ends.

[0064] [Activity A15] On the other hand, if the fitness meets the update conditions (in this embodiment, if the fitness is below the threshold), the process proceeds to activity A15, and the processor 23 notifies the user of information regarding the update of the learning model M1.

[0065] [Activity A16] Next, the process proceeds to activity A16, and processor 23 requests a stop notification from production equipment 4. The stop notification is information indicating that a predetermined stop operation has been performed at production equipment 4, which is currently in operation. In response to this request, a notification is sent to the on-site personnel operating production equipment 4, instructing them to perform the stop operation.

[0066] [Activity A17] Subsequently, a stop notification is generated when a stop operation is performed by the on-site staff. When the acquisition unit 231 acquires the stop notification, the process proceeds to activity A17, and the update unit 236 updates the learning model. In other words, the update unit 236 updates the learning model if the fitness calculated based on the operational information IF2 satisfies predetermined update conditions. The production equipment 4 is then put back into operation using the updated learning model M1.

[0067] In this embodiment, the update unit 236 further updates the layout design of the virtual production equipment 5 based on observation results from the detection device 453, etc., when the fitness level satisfies predetermined update conditions. This allows the virtual production equipment 5 to reflect the latest operating status of the production equipment 4, etc., even if there is a discrepancy between the operating status of the virtual production equipment 5 and the operating status of the production equipment 4 in the real space, due to circumstances such as the virtual production equipment 5 not being able to keep up with changes in the status of the production equipment 4 in the real space. Therefore, the discrepancy can be reduced. For example, when the detection device 453 detects new items, etc., the processor 23 generates an object corresponding to the items, etc., at the corresponding location in the virtual production equipment 5 based on the image data generated by the detection device 453. Also, when the detection device 453 observes that existing equipment, such as the receiving equipment 41, etc., has moved from its position based on the initial design information IF1, the processor 23 updates the position of the virtual receiving equipment 51 in the virtual production equipment 5 based on the observation results.

[0068] After the processing of Activity A17, the first information processing is completed.

[0069] Through the information processing described above, equipment implemented or planned to be implemented in the real world can be reproduced as virtual equipment in a virtual space based on design information that shows the layout design. This allows users to evaluate the validity of the equipment based on the results of physical simulations performed in the virtual space, before actually constructing or operating the equipment, or when actually operating the equipment. Therefore, the cost of constructing or operating the equipment can be reduced.

[0070] 3.2. Regarding the second information processing flow Next, we will explain the second information processing flow. Figure 8 is an activity diagram showing an example of the second information processing flow executed in the information processing system 1.

[0071] [Activity A101] First, in activity A101, the acquisition unit 231 acquires receiving information regarding the arrival of products. As a result, the processor 23 recognizes the existence of products that require handling.

[0072] [Activity A102] Next, the process proceeds to activity A102, where processor 23 instructs forklift 45 to begin loading.

[0073] [Activity A103] Next, the process proceeds to activity A103, where the forklift 45 performs self-position estimation to determine its own position on the occupied grid map. The method of self-position estimation is arbitrary, including methods using external signals such as GPS, and methods using landmark information obtained by observing predetermined landmarks with the detection device 453. In this embodiment, the forklift 45 performs self-position estimation based on information detected in real space. This makes it possible to suppress the operation of the forklift 45 when the fitness between the production equipment 4 and the virtual production equipment 5 is low. Landmark detection can be achieved by training the learning model M1, etc., to identify landmarks and using it as a landmark classifier.

[0074] [Activity A104] Next, the process proceeds to activity A104, where the acquisition unit 231 acquires the estimated self-position of the forklift 45 and generates a receiving route based on the generated virtual production facility 5. The receiving route is the route R from the starting position (i.e., the estimated self-position of the forklift 45) to the receiving facility 41 (specifically, the location where the received products are placed).

[0075] [Activity A105] Next, the process proceeds to activity A105, and the forklift 45 moves along the transmitted receiving route to the receiving equipment, which is the target location. At this time, the forklift 45 updates the occupied grid map based on its own position and the observation results of the detection device 453.

[0076] [Activity A106] Next, the process proceeds to activity A106, and the forklift 45 transmits various sensor information to the processor 23 during its movement from the starting position to the receiving equipment 41. The sensor information includes the detection results of the detection device 453 (e.g., continuous images, image data captured according to a predetermined frame rate, etc.), the detection results of the forklift 45's speedometer, and the forklift 45's position information. In other words, the forklift 45 transmits its movement history as sensor information to the processor 23. At this time, the sensor information is transmitted in association with the operation information IF2. For example, the sensor information associates the type and time of operation of the production equipment 4 specified in the operation information IF2 with the operation and time of the production equipment 4 (specifically the forklift 45) actually detected by the detection device 453, etc. The difference between these tends to increase as the discrepancy between the production equipment 4 and the virtual production equipment 5 increases.

[0077] [Activity A107] Next, the process proceeds to activity A107, where the processor 23 acquires the sensor information transmitted in activity A106 and records it in the storage unit 22, etc. In particular, the processor 23 acquires information regarding the recovery operation performed when the stop operation described later occurs as sensor information and stores it in the storage unit 22, etc.

[0078] [Activity A108] Subsequently, when the forklift 45 reaches the receiving facility 41, the process proceeds to activity A108, and the forklift 45 performs cargo handling. This causes the forklift 45 to load the cargo to be handled, which is located at the receiving facility 41, onto the cargo handling device 452 and move it to a predetermined location (for example, the outbound facility 42 or storage facility 43). The cargo handling operation involves, for example, detecting the cargo to be handled by inputting the observation results of the cargo to be handled using the detection device 453 into the learning model M1, guiding the cargo to be handled (for example, adjusting the relative position between the forks of the cargo handling device 452 and the pallet, adjusting the angle difference between the forks and the insertion holes of the pallet, etc.), and loading the cargo to be handled by the operation of the cargo handling device 452 (for example, raising and lowering the forks, etc.). Once it is confirmed that the cargo to be handled has been loaded onto the cargo handling device 452, the forklift 45 terminates the cargo handling process.

[0079] [Activity A109] Next, the process proceeds to activity A109, where the forklift 45 transmits sensor information from the cargo handling process performed in activity A108 to the processor 23, similar to activity A106.

[0080] [Activity A110] Next, the process proceeds to activity A110, where the processor 23, similar to activity A107, acquires the sensor information transmitted in activity A109 and records it in the storage unit 22, etc.

[0081] [Pier A111] Next, the process proceeds to activity A111, where the generation unit 232 generates an outbound route from the end position of the cargo handling process to the outbound equipment 42. The method of generating the outbound route is the same as that of the route R or inbound route described above. The processor 23 then transmits the generated outbound route to the forklift 45.

[0082] [Activity A112] Next, the process proceeds to activity A112, where the forklift 45 acquires the generated outbound route and moves along the outbound route to the outbound equipment 42. At this time, the forklift 45 updates the occupied grid map based on its own position and the observation results of the detection device 453.

[0083] [Pier A113] Next, the process proceeds to activity A113, and the forklift 45 transmits sensor information from the cargo handling process performed in activity A108 to the processor 23, similar to activity A106.

[0084] [Activity A114] Next, the process proceeds to activity A114, where the processor 23, similar to activity A107, acquires the sensor information transmitted in activity A109 and records it in the storage unit 22, etc.

[0085] [Pilot A115] Next, the process proceeds to activity A115, and the forklift 45 performs an unloading operation. This causes the cargo to be unloaded into the outbound equipment 42.

[0086] [Activity A116] Next, the process proceeds to activity A116, where the forklift 45 transmits sensor information from the cargo handling process performed in activity A108 to the processor 23, similar to activity A106.

[0087] [Activity A117] Next, the process proceeds to activity A117, where the acquisition unit 231 acquires the sensor information transmitted in activity A109, similar to activity A107, and records it in the storage unit 22, etc.

[0088] [Activity A118] Next, the process proceeds to activity A118, and the forklift 45 notifies the processor 23 that the handling is complete. Subsequently, the handled items are released from the production equipment 4.

[0089] [Activity A119] Next, the process proceeds to activity A119, where the processor 23 executes the fitness calculation process. Based on the acquired sensor information, the map information of production equipment 4 (in this embodiment, the occupied grid map in real space), and the map information of virtual production equipment 5 (in this embodiment, the virtual occupied grid map in virtual space), the processor 23 calculates the fitness of the learning model M1 and records it in the memory unit 22, etc. Here, we will explain the case where the fitness is calculated based on the similarity of the occupied grid maps.

[0090] First, the processor 23 refers to the memory unit 22 and other components to acquire sensor information, particularly reset operations, including the operation history of the forklift 45. Next, based on the acquired sensor information, the processor 23 acquires the latest occupied grid map in real space, compares this occupied grid map with a virtual occupied grid map in the virtual production facility 5, and calculates the similarity. At this time, the processor 23 may set the grid weights for calculating the similarity based on the acquired sensor information. For example, the processor 23 may set the weight of the grid corresponding to the position where a reset operation was performed on the occupied grid map to be greater than the weights of other grids. Emergency operations such as stopping operations and reset operations are correlated with the discrepancy between the production facility 4 and the virtual production facility 5. Therefore, the discrepancy between the production facility 4 and the virtual production facility 5 that arises from such operations can be reflected in the fitness.

[0091] Through this second information processing, a low fitness record is generated, and the processor 23 performs the first information processing while referring to the low fitness record thus generated.

[0092] 3.3. Processing flow when a stop operation is performed Next, we will explain the processing flow when a stop operation is performed during the second information processing. Figure 9 is an activity diagram showing an example of the processing flow when a stop operation is performed in the information processing system 1.

[0093] [Activity A201] Next, the process proceeds to activity A201, where the forklift 45 receives a stop command from the user (e.g., the operator or supervisor of the forklift 45). The stop command includes, for example, manual braking or pressing an emergency stop button, which are not scheduled for the forklift 45's automatic operation, to bring it to an emergency stop. The stop command may also involve the generation of a signal for the forklift 45 to spontaneously issue a stop command when the detection device 453 detects an obstacle not recorded in the occupied grid map and determines that the obstacle obstructs travel along the path R.

[0094] [Activity A202] Next, the process proceeds to activity A202, and forklift 45 performs a stop operation. As a result, forklift 45 stops moving and comes to a temporary halt in place. At this time, forklift 45 is in a state where its movement is restricted.

[0095] [Activity A203] Next, the process proceeds to activity A203, where the forklift 45 accepts a predetermined return operation from the user.

[0096] [Activity A204] Next, the process proceeds to activity A204, where forklift 45 performs a recovery process in response to a recovery command from the user. This makes forklift 45 ready to operate again.

[0097] [Activity A205] Next, the process proceeds to activity A205, and the forklift 45 resumes loading and unloading operations. At this time, the information processing system 1 may resume processing from the point when the stop operation was performed, or it may perform the second information processing described above from the beginning. At this time, the forklift 45 transmits operation information related to the stop operation and the return operation to the processor 23, linked to the sensor information.

[0098] [Activity A206] Next, the process proceeds to activity A206, where the processor 23 acquires sensor information and operation information and records it in the storage unit 22, etc.

[0099] This third information processing method allows the correlation between sensor information and operation information to be recorded in real time in conjunction with the operation of production equipment 4. This correlation is used, for example, when calculating fitness in the fitness calculation process of activity A119.

[0100] In the above embodiment, the first, second, and third information processing steps were described using separate activity diagrams. However, these steps may be performed individually or in conjunction with each other. Furthermore, these steps may be performed independently or in parallel.

[0101] 4. Others The above-described method of information processing is merely an example and is not limited to it.

[0102] The equipment to which the above information processing is applied is not limited to production equipment 4 such as the shared warehouse described above. For example, the above information processing may be applied to machine equipment that recognizes a cutting target based on the learning model M1 and performs machining on the target according to the recognition result, or it may be applied to placement equipment that recognizes product specifications based on the learning model M1 and arranges products on a robot arm as a drive device according to the recognition result.

[0103] Figure 10 shows an example of a machine tool 6. The machine tool 6 includes a cutting device 61 as a drive device and a holding device 62. The cutting device 61 is configured to process a workpiece W held by the holding device 62 and is configured to be movable in two or three dimensions based on control commands from the processor 23. The processor 23, for example, as an acquisition unit 231, acquires design information IF1 indicating the layout design of the cutting device 61 and the holding device 62 (e.g., the positional relationship, dimensions, material, etc. of the cutting device 61, the holding device 62, and the workpiece W) and operation information IF2 regarding the operation status of the machine tool 6 in real space implemented according to the layout design (e.g., the cutting sequence of the workpiece W, the number of cuts of the workpiece W, etc.). Based on this information, a virtual machine tool 7 corresponding to the machine tool 6 in virtual space is generated. The virtual machine tool 7 is an object corresponding to the cutting device 61, and the virtual holding device 72 is an object corresponding to the holding device 62. The processor 23 performs a physical simulation in the virtual machine tool 7, thereby cutting a virtual cutting target W in the virtual machine tool 7 and generates image data of the cutting target W that conforms to the specifications as training data D1. At this time, the processor 23 generates image data of the cutting target W that does not conform to the specifications, so-called defective products, as disturbance training data D2, based on disturbance information IF3, which includes dimensional tolerances and fluctuations in the operation of the virtual cutting device 71. The learning unit 235 uses the dataset containing this training data D1 and disturbance training data D2 to train the learning model M1, and based on the results of this training, outputs the learning parameters of the learning model M1 for the virtual machine tool 7 in the real space and updates the learning model M1.

[0104] Disturbance information IF3 is not limited to information about light emitted or introduced from the virtual lighting equipment 54, i.e., light source information. For example, disturbance information IF3 may also include information about disturbances other than light, such as sound, vibration, and heat, generated by the operation of each object included in the virtual production equipment 5.

[0105] The learning model M1 may be the same for both real-world and virtual spaces.

[0106] The information processing device 2 may be on-premise or in a cloud-based configuration. In the case of a cloud-based information processing device 2, for example, the above-mentioned functions and processing may be provided in the form of SaaS (Software as a Service) or cloud computing.

[0107] In the above embodiment, the information processing device 2 performed various storage and control functions, but instead of the information processing device 2, multiple external devices may be used. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like.

[0108] The above embodiment is not limited to the information processing system 1, but may also be an information processing method or an information processing program. The information processing method includes each step of the information processing system 1. The information processing program causes at least one computer to execute each step of the information processing system 1.

[0109] The above-mentioned information processing system 1, etc., may be provided in any of the following embodiments.

[0110] (1) An information processing system comprising at least one processor capable of executing a program such that the following steps are performed, wherein in the acquisition step, design information indicating the layout design of equipment and operation information relating to the operational status of equipment in real space implemented according to the layout design, wherein the equipment includes a drive device that can be driven based on a learning model including at least one learning parameter, in the first generation step, a virtual equipment is generated based on the design information and the operation information, wherein the virtual equipment is the equipment reproduced in a predetermined virtual space and configured to be operational in the virtual space by the learning model, in the second generation step, at least one learning data is generated based on the results of a predetermined physical simulation of the virtual equipment, wherein the learning data is used to train the learning model of the virtual equipment, and in the output step, the learning parameter of the learning model in real space is output based on the results of training the learning model of the virtual equipment using the learning data.

[0111] With this configuration, equipment implemented or planned to be implemented in the real world can be reproduced as virtual equipment in a virtual space based on design information that shows the layout design. This allows users to evaluate the validity of the equipment based on the results of physical simulations performed in the virtual space, before actually constructing or operating the equipment, or when actually operating the equipment. Therefore, the cost of constructing or operating the equipment can be reduced.

[0112] (2) An information processing system as described in (1) above, wherein the acquisition step further includes disturbance information relating to disturbances in the equipment, the second generation step further generates disturbance learning data by applying the disturbance to at least one of the learning data based on the disturbance information in the virtual space, and the output step further outputs the learning parameters of the learning model in the real space based on the results of learning the learning model of the virtual equipment using the disturbance learning data.

[0113] With this configuration, changes in real-world conditions, such as lighting degradation, the introduction of new equipment, weather, other pedestrians, other vehicles, and robots, are incorporated as disturbances in the physical simulation, allowing for the acquisition of learning parameters that are more applicable to a wider range of situations. Therefore, the operational efficiency of the equipment can be improved.

[0114] (3) An information processing system according to (1) or (2) above, wherein the physical simulation includes an optical simulation, and the learning data includes at least one of the image data and LiDAR data of the virtual facility.

[0115] With this configuration, comparing the observation results of the equipment by users involved in its operation with the visual information of the virtual equipment makes it easier to intuitively judge the validity of the learning results.

[0116] (4) An information processing system according to any one of (1) to (3) above, wherein in the learning step, at least the learning model of the virtual equipment based on the learning data is learned.

[0117] (5) An information processing system described in any one of (1) to (4) above, wherein in the update step, the learning model is updated when the fitness calculated based on the operational information satisfies predetermined update conditions, and the fitness indicates the consistency between the operational status of the virtual equipment and the operational status of the equipment in the real space.

[0118] With this configuration, for example, if a discrepancy arises between the operational status of the virtual equipment and the operational status of the equipment in the real world due to circumstances such as the virtual equipment being unable to keep up with changes in the status of the equipment in the real world, the learning model is updated. This makes it easier to keep the discrepancy within an acceptable range.

[0119] (6) An information processing system described in any one of (1) to (5) above, wherein in the evaluation step, an evaluation index indicating the validity of the design information is output based on the physical simulation, and in the learning stop step, learning of the learning model is not performed if the evaluation index satisfies a predetermined stop condition.

[0120] With this configuration, if the learning assumptions are not valid due to circumstances such as the physical layout of the equipment being difficult to implement, the learning model will not be trained. This reduces wasted computing resources.

[0121] (7) An information processing system as described in (6) above, wherein in the presentation step, when the stop condition is met, the information processing system presents stop information relating to the met stop condition in a manner that is visible to the user.

[0122] With this configuration, users can efficiently review design information based on shutdown information.

[0123] (8) An information processing method comprising each step of the information processing system described in any one of (1) to (7) above.

[0124] (9) An information processing program that causes at least one computer to perform each step of the information processing system described in any one of (1) to (7) above. Of course, this is not always the case.

[0125] Finally, while various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0126] 1: Information Processing System 2: Information Processing Device 20: Communications bus 21: Communications Department 22: Storage section 23: Processor 231: Acquisition Department 232 :Generation part 233: Simulation Department 234: Evaluation Department 235: Learning Department 236: Update section 237:Presentation part 238: Learning Stop Unit 3: User terminal 30: Communications bus 31: Communications Department 32: Storage section 33: Processor 34:Display section 35: HMI devices 4: Production equipment 41: Warehousing equipment 42: Dispatch equipment 43:Storage equipment 44: Lighting equipment 45: Forklift 451: Vehicle body 452: Cargo handling equipment 453: Detection device 5: Virtual production facilities 51: Virtual storage facility 52: Virtual Inventory Dispatch Equipment 53: Virtual storage facilities 54: Virtual lighting equipment 55: Virtual Forklift 551: Virtual vehicle 552: Virtual cargo handling equipment 553: Virtual Detector 6:Work equipment 61: Cutting equipment 62: Holding device 7: Virtual Workshop Equipment 71: Virtual cutting machine 72: Virtual storage device D1: Training data D2: Disturbance training data P1: Waypoint R: Route W: Cutting target

Claims

1. An information processing system, The system comprises at least one processor capable of executing a program so that each of the following steps is performed, In the acquisition step, design information indicating the layout design of the equipment, operational information regarding the operational status of the equipment in the real space implemented according to the layout design, and disturbance information regarding disturbances in the equipment are acquired, and here, The equipment comprises a drive device that can be driven based on a classifier including at least one learning parameter, and an imaging unit that can image an object handled by the equipment. The identifier is configured to be able to identify the object, The drive device is configured to perform operations on the object based on the imaging results of the imaging unit and according to the object identification result of the identifier, The field of view of the imaging unit changes according to the layout of the equipment and the operating status of the equipment. The disturbance includes light disturbances from light sources in the equipment. The light source changes the way the layout of the equipment is viewed by the imaging unit. The aforementioned operational information includes operations relating to the irradiation pattern of light from the light source by the equipment, In the first generation step, a virtual facility is generated, which is the facility reproduced in a predetermined virtual space, based on the design information and the operation information. The virtual equipment comprises a virtual drive unit that reproduces the drive unit in the virtual space, and a virtual imaging unit that reproduces the imaging unit in the virtual space. In the second generation step, at least one training data is generated based on the results of a predetermined physical simulation that operates the virtual equipment based on the operational information, wherein the training data is used to train the classifier of the virtual equipment and includes the imaging results of a virtual object reproduced in the virtual space by the virtual imaging unit and the correct label of the virtual object. In the output step, based on the learning results of the classifier of the virtual equipment using the training data, the training parameters of the classifier in the real space are output. In the second generation step, based on the disturbance information in the virtual space, the light from the light source is applied to the imaging results of the virtual object included in the learning data in the virtual space, thereby generating disturbance learning data in which the disturbance of light is applied to the imaging results by the virtual imaging unit. The information processing system further outputs the learning parameters of the classifier in the real space based on the learning results of the classifier of the virtual facility using the disturbance learning data in the output step.

2. In the information processing system described in claim 1, The aforementioned physical simulation includes optical simulations, The learning data includes at least one of the image data and LiDAR data of the virtual facility, in an information processing system.

3. In the information processing system described in claim 1, Furthermore, the information processing system performs training on the virtual equipment classifier based on the training data in the learning step.

4. In the information processing system described in claim 1, Furthermore, in the update step, the classifier is updated if the fitness calculated based on the operational information satisfies predetermined update conditions, and here the fitness indicates the consistency between the operational status of the virtual equipment and the operational status of the equipment in the real space, according to the information processing system.

5. In the information processing system described in claim 1, In the evaluation step, evaluation indicators showing the validity of the design information are output based on the physical simulation. In the learning stop step, the information processing system does not perform learning of the classifier if the evaluation indicator satisfies a predetermined stopping condition.

6. In the information processing system described in Claim 5, The presentation step involves an information processing system that, when the aforementioned stop conditions are met, presents stop information relating to the met stop conditions in a manner that is visible to the user.

7. An information processing method performed by an information processing system, An information processing method comprising each step of the information processing system described in any one of claims 1 to 6.

8. It is an information processing program, An information processing program that causes at least one computer to perform each step of the information processing system described in any one of claims 1 to 6.

Images