Machine learning execution program, data conversion program, machine learning execution device, data conversion device, machine learning execution method, and data conversion method

Abstract

To accurately convert a handwritten drawing into a data format that can be handled on a computer-aided design system.SOLUTION: A data conversion program causes a computer to execute: acquiring, as inference data, handwritten drawing data for inference indicating a handwritten drawing for inference relating to a design object for inference; identifying an area where elements that constitute the design object are drawn in the handwritten drawing relating to the design object; outputting detailed data relating to the inference data by inputting the inference data into a machine learning model including an element data output model that executes, for each element, processing for outputting element data indicating information for identifying a type of the element and a position of the area and a detailed data output model that executes, for each element, processing for outputting detailed data indicating information related to the element based on the element data, data-converting the handwritten drawing for inference indicated by the handwritten drawing data for inference based on the detailed data, and outputting a data converted drawing.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a machine learning execution program, a digitization program, a machine learning execution device, a digitization device, a machine learning execution method, and a digitization method. [Background technology]

[0002] When designing and manufacturing industrial products, hand-drawn drawings of the industrial products must be converted into a data format that can be handled by a computer-aided design (CAD) system. For example, when building a house, architects and designers create hand-drawn drawings of the house in consultation with the client, and then convert the hand-drawn drawings into a data format that can be handled by the CAD system before construction begins. Furthermore, when changes are made to the house design, the changes are manually written into the drawings, which are in a data format that can be handled by the CAD system, and then converted back into the aforementioned data format.

[0003] One example of a technique used in such a situation is the construction drawing recognition device disclosed in Patent Document 1. This construction drawing recognition device has an image data input means, an expansion means, a dot count measurement array data creation means, and a contour / skeleton recognition means. The image data input means inputs image data obtained by scanning an image of a construction drawing. The expansion means expands the contours of portions of the image data input by said means that consist of black or white dots. The dot count measurement array data creation means measures the number of black or white dots in the horizontal and vertical directions of the image data whose contours have been expanded by said means, and creates horizontal and vertical dot count measurement array data. The contour / skeleton recognition means recognizes the contour and skeleton of the construction drawing based on the dot count measurement array data in both directions created by said means. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 09-128425 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned construction drawing recognition device is limited in the information that can be identified. In addition, it can be difficult to set the feature values of the algorithm, and if the feature values deviate from the expected range, the accuracy of recognizing the outline and skeleton of the construction drawing can decrease.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a machine learning execution program, a digitization program, a machine learning execution device, a digitization device, a machine learning execution method, and a digitization method that can accurately convert hand-drawn drawings into a data format that can be handled on a computer-aided design system. [Means for solving the problem]

[0007] According to one aspect of the present invention, a computer is provided with: a learning data acquisition function for acquiring learning data including hand-drawn drawing data representing a drawing relating to a design object and including hand-drawn shapes; an element acquisition function for determining the number of times of processing for classifying elements indicated by the element data in order to output detailed data which is information indicating elements constituting the design object; In the handwritten drawing shown by the handwritten drawing data, The aforementioned an element data output model that identifies an area in which an element is drawn and executes, for each element, a process of outputting information that identifies the type of the element and element data that indicates the position of the area; and a detailed data output model that executes, for each element, a process of outputting detailed data that indicates information about the element based on the element data. a detailed data output model having a model structure based on the determination result of the number of times of processing; and 、 and a machine learning execution program that realizes a machine learning execution function of inputting the learning data into a machine learning model including the learning data and updating the machine learning model.

[0008] In the above-described machine learning execution program, when the machine learning execution function only needs to execute the process of classifying the elements indicated by the element data once to output the detailed data, the learning data may be input to the machine learning model including the detailed data output model that executes the process of outputting the detailed data including member data indicating information for identifying a member that realizes the element and direction data indicating the direction in which the member is attached to the design target.

[0009] In the above-described machine learning execution program, when the machine learning execution function needs to execute the process of classifying the elements indicated by the element data two or more times to output the detailed data, the learning data may be input to the machine learning model including the detailed data output model including a member identification model that outputs member data indicating information for identifying a member that realizes the element based on the element data and a direction identification model that outputs direction data indicating the direction in which the member is attached to the design target based on the member data.

[0010] In addition, the above-described machine learning execution program may further cause the computer to realize a post-processing function that extracts at least a part of the contour of the region based on at least one of the element data and the detailed data and outputs post-processed data indicating at least a part of the contour of the region.

[0011] In the above-described machine learning execution program, when the machine learning execution function outputs at least one of the element data output model that outputs, based on the handwritten drawing data, the deletion elements to be deleted from the handwritten drawing indicated by the handwritten drawing data and the element data indicating the positions of the regions where the deletion elements are drawn, and the addition elements to be added to the handwritten drawing indicated by the handwritten drawing data and the element data indicating the positions of the regions where the addition elements are drawn, the learning data may be input to the machine learning model including the element data output model.

[0012] One aspect of the present invention is to cause a computer to have an inference data acquisition function for acquiring, as inference data, inference handwritten drawing data indicating an inference handwritten drawing that is a drawing related to a design object for inference and in which a handwritten figure is drawn, an element acquisition function that determines the number of times of processing to classify elements indicated by element data in order to output detailed data that is information indicating elements that constitute the inference design object; in a handwritten drawing that is a drawing related to a design object and in which a handwritten figure is drawn The aforementioned an element data output model that executes, for each element, a process of specifying a region in which the element is drawn and outputting element data indicating information specifying the type of the element and the position of the region, and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information related to the element based on the element data a detailed data output model having a model structure based on the determination result of the number of times of processing; a detailed data output function that inputs the inference data into a machine learning model including the above and causes the machine learning model to output the detailed data related to the inference data, and a data conversion function that converts the inference handwritten drawing indicated by the inference handwritten drawing data based on the detailed data related to the inference data and outputs a converted drawing.

[0013] One aspect of the present invention is a learning data acquisition unit that acquires learning data including handwritten drawing data indicating a handwritten drawing that is a drawing related to a design object and in which a handwritten figure is drawn, an element acquisition function for determining the number of times of processing for classifying elements indicated by the element data in order to output detailed data which is information indicating elements constituting the design object; in the handwritten drawing indicated by the handwritten drawing data The aforementioned an element data output model that executes, for each element, a process of specifying a region in which the element is drawn and outputting element data indicating information specifying the type of the element and the position of the region, and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information related to the element based on the element data a detailed data output model having a model structure based on the determination result of the number of times of processing; and 、 a machine learning execution unit that inputs the learning data into a machine learning model including the above and updates the machine learning model.

[0014] One aspect of the present invention is an inference data acquisition unit that acquires, as inference data, inference handwritten drawing data indicating an inference handwritten drawing that is a drawing related to a design object for inference and in which a handwritten figure is drawn, an element acquisition unit that determines the number of times of processing to classify elements indicated by element data in order to output detailed data that is information indicating elements that constitute the inference design object;A handwritten drawing which is a drawing related to a design object and has handwritten figures drawn thereon The aforementioned An element data output model that identifies a region where an element is drawn and executes, for each element, a process of outputting information identifying the type of the element and element data indicating the position of the region, and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information related to the element based on the element data a detailed data output model having a model structure based on the determination result of the number of times of processing; and 、 A detailed data output unit that inputs the inference data into a machine learning model including the above and causes the machine learning model to output the detailed data related to the inference data, and a data conversion unit that converts the inference handwritten drawing data represented by the inference handwritten drawing based on the detailed data related to the inference data and outputs a converted drawing. A data conversion device comprising:

[0015] One aspect of the present invention is that an input unit of a computer acquires learning data including handwritten drawing data which is a drawing related to a design object and shows a handwritten drawing with handwritten figures drawn thereon, determining the number of times of processing for classifying elements indicated by element data in order to output detailed data which is information indicating elements constituting the design object; in the handwritten drawing shown by the handwritten drawing data, an operation unit of the computer The aforementioned identifies a region where an element is drawn, and an element data output model that executes, for each element, a process of outputting information identifying the type of the element and element data indicating the position of the region, and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information related to the element based on the element data a detailed data output model having a model structure based on the determination result of the number of times of processing; and 、 A machine learning execution method of inputting the learning data into a machine learning model stored in a storage unit of the computer and updating the machine learning model

[0016] One aspect of the present invention is that an input unit of a computer acquires, as inference data, inference handwritten drawing data which is a drawing related to an inference design object and shows an inference handwritten drawing with handwritten figures drawn thereon, an element acquisition unit of the computer determines the number of times of processing for classifying elements indicated by element data in order to output detailed data which is information indicating elements constituting the inference-use design object; in a handwritten drawing which is a drawing related to a design object and has handwritten figures drawn thereon, an operation unit of the computer The aforementionedan element data output model that identifies an area in which an element is drawn and executes, for each element, a process of outputting information that identifies the type of the element and element data that indicates the position of the area; and a detailed data output model that executes, for each element, a process of outputting detailed data that indicates information about the element based on the element data. a detailed data output model having a model structure based on the determination result of the number of times of processing; and 、 the inference data is input to a machine learning model stored in a memory unit of the computer, causing the machine learning model to output the detailed data related to the inference data, and an output unit of the computer digitizes the hand-drawn drawing for inference indicated by the hand-drawn drawing data for inference based on the detailed data related to the inference data, and outputs a digitized drawing. [Effects of the Invention]

[0017] According to the present invention, a hand-drawn drawing can be converted with high accuracy into a data format that can be handled on a computer-aided design system. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of a machine learning execution program according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a handwritten drawing according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of models included in a machine learning model according to an embodiment, and an example of inputs and outputs of each model. [Figure 4] FIG. 3 is a diagram showing an example of an image obtained by using semantic segmentation to identify an area in the handwritten drawing shown in FIG. 2 where a wall or fixture is depicted. [Figure 5] FIG. 3 is a diagram showing an example of an image obtained by using semantic segmentation to identify the region in which a box is drawn in the handwritten drawing shown in FIG. 2. [Figure 6]FIG. 3 is a diagram showing an example of an image obtained by using semantic segmentation to identify the area of the hand-drawn drawing shown in FIG. 2 where floor slabs are drawn. [Figure 7] FIG. 3 is a diagram showing an example of an image obtained by using semantic segmentation to identify the area surrounded by the exterior wall lines in the hand-drawn drawing shown in FIG. 2. [Figure 8] FIG. 3 is a diagram showing an example of an image obtained by using semantic segmentation to identify the areas in which each room is drawn in the handwritten drawing shown in FIG. 2. [Figure 9] FIG. 5 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 4. [Figure 10] FIG. 6 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 5. [Figure 11] FIG. 7 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 6. [Figure 12] FIG. 8 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 7. [Figure 13] FIG. 9 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 8. [Figure 14] FIG. 10 is a diagram showing an example of a digitized drawing included as an answer in learning data according to an embodiment. [Figure 15] 10 is a flowchart illustrating an example of processing executed by a machine learning execution program according to an embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of processing executed by the machine learning execution program according to the embodiment when correcting a handwritten drawing. [Figure 17] FIG. 2 is a diagram illustrating an example of a functional configuration of a data conversion program according to an embodiment. [Figure 18] FIG. 1 is a diagram showing an example of a drawing that can be handled on a computer-aided design system. [Figure 19] FIG. 19 is a diagram showing an example of a drawing obtained by manually correcting the drawing shown in FIG. 18. [Figure 20] FIG. 2 is a diagram showing an example of a digitized drawing according to the embodiment. [Figure 21] FIG. 2 is a diagram illustrating an example of a tree structure of elements constituting an inference-use design object according to the embodiment. [Figure 22] 10 is a flowchart illustrating an example of processing executed by a data conversion program according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] [Embodiment] An embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the functional configuration of a machine learning execution program according to an embodiment. As shown in Fig. 1, a machine learning execution program 10 includes a learning data acquisition function 11, a machine learning execution function 12, and a post-processing function 13.

[0020] At least some of the functions of the machine learning execution program 10 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) reading and executing the machine learning execution program 10 stored in a storage device. The storage device referred to here is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or a read-only memory (ROM).

[0021] The learning data acquisition function 11 acquires learning data including handwritten drawing data representing drawings related to the design of the design object, which are handwritten drawings containing handwritten shapes. The handwritten drawings referred to here may be entirely handwritten drawings or drawings that have been handwritten on drawings created by a computer-aided design system or the like. The handwritten drawings referred to here include writing with a pen, pencil, or the like on a drawing drawn on paper, and also include writing using a drawing function on a drawing displayed on a touch panel display mounted on a tablet or the like. The design object may be, for example, a building such as a house.

[0022] FIG. 2 is a diagram showing an example of a handwritten drawing according to an embodiment. For example, the learning data acquisition function 11 acquires learning data including handwritten drawing data representing the handwritten drawing shown in FIG. 2. The handwritten drawing data is, for example, raster data. The handwritten drawing may be a color drawing or a grayscale drawing. However, when the handwritten drawing is a color drawing, converting it to a grayscale drawing often improves the accuracy of processing by the machine learning execution function 12. An example of the process of converting a color drawing to a grayscale drawing is a process of multiplying the red, green, and blue scales assigned to each pixel of the color drawing by appropriate coefficients.

[0023] Furthermore, before processing by the machine learning execution function 12, it is preferable that noise that may reduce the accuracy of learning by the machine learning model 120 described below is removed from the grayscale drawing by at least one of binarization and morphological transformation. Such noise includes, for example, highlighter markings and ruled lines in the background of the drawing. Furthermore, highlighter markings often vary in color, thickness, etc. depending on the architect, designer, etc., and therefore are likely to reduce the accuracy of learning by the machine learning model 120 described below.

[0024] In addition, when a building wall is drawn on the handwritten drawing, the learning data may include, as an answer, the area where the wall is drawn. Further, it is preferable that the width of the area is wider than the thickness of the wall. Examples of the width of such an area include a width three times the thickness of the wall. When the learning data includes, as an answer, an area having a width wider than the thickness of the wall, even if there is some error in the scale of the handwritten drawing, the machine learning model 120 described later can recognize the answer. When the learning data includes, as an answer, an area having a width wider than the thickness of the wall, the learning accuracy of the machine learning model 120 described later can be improved compared to the case where the learning data includes only an area having a width equal to or less than the thickness of the wall.

[0025] The machine learning execution function 12 inputs the learning data into the machine learning model 120 shown in FIG. 3 to update the machine learning model 120. FIG. 3 is a diagram showing an example of the models included in the machine learning model according to the embodiment and the inputs and outputs of each model. As shown in FIG. 3, the machine learning model 120 includes an element data output model 121, an element acquisition function 122, a detailed data output model 123, an element acquisition function 124, a member identification model 125, a member acquisition function 126, and a direction identification model 127.

[0026] The element data output model 121 acquires the handwritten drawing data acquired by the learning data acquisition function 11. Then, for each element, the element data output model 121 executes a process of identifying the area where the element constituting the design target is drawn in the handwritten drawing indicated by the handwritten drawing data, and outputs element data indicating the type of the element and the position of the area.

[0027] The elements mentioned here are, for example, walls, furniture, boxes, floor slabs, and rooms. Also, the types of elements mentioned here are, for example, the manufacturer, seller, and model number of the members that realize the elements. Furthermore, it is preferable that the elements are classified into a certain number of categories or more by the name of the member or the like rather than being classified in detail by the manufacturer, seller, model number, etc. of the member. For example, the five elements of "entrance storage tall type", "entrance storage float type", "entrance storage full tall type", "entrance storage wall type", and "entrance storage floor type" are preferably classified into one category of "entrance storage" rather than being classified into separate categories because the accuracy of the processing by the machine learning model 120 is improved.

[0028] Also, when the element data output model 121 executes such processing, for example, it applies semantic segmentation to the handwritten drawing indicated by the handwritten drawing data. Semantic segmentation gives the meaning such as the type and name of the object drawn in the area including each pixel of the image of the handwritten drawing to each pixel.

[0029] FIG. 4 is a diagram showing an example of an image obtained by specifying the area in which a wall or furniture is drawn in the handwritten drawing shown in FIG. 2 using semantic segmentation. In FIG. 4, the area where the wall is drawn is hatched with dots, and the area where the furniture is drawn is hatched with oblique lines. For example, the element data output model 121 applies semantic segmentation to the handwritten drawing shown in FIG. 2 and specifies the area where the wall is drawn in the handwritten drawing. Then, the element data output model 121 outputs element data indicating the information for specifying the type of the wall and the position of the area indicated by the dot hatching in FIG. 4.

[0030] FIG. 5 is a diagram showing an example of an image obtained by identifying, using semantic segmentation, a region in the handwritten drawing shown in FIG. 2 where a box is drawn. In FIG. 5, six regions where boxes are drawn are hatched with different hatchings. For example, the element data output model 121 applies semantic segmentation to the handwritten drawing shown in FIG. 2 to identify a region in the handwritten drawing where a box is drawn. Then, the element data output model 121 outputs element data indicating information for identifying the type of the box and the position of the region indicated by dot hatching in FIG. 5.

[0031] FIG. 6 is a diagram showing an example of an image obtained by identifying, using semantic segmentation, a region in the handwritten drawing shown in FIG. 2 where a floor slab is drawn. In FIG. 6, the region where the floor slab is drawn is hatched. For example, the element data output model 121 applies semantic segmentation to the handwritten drawing shown in FIG. 2 to identify a region in the handwritten drawing where a floor slab is drawn. Then, the element data output model 121 outputs element data indicating information for identifying the type of the floor slab and the position of the region indicated by dot hatching in FIG. 6.

[0032] FIG. 7 is a diagram showing an example of an image obtained by identifying, using semantic segmentation, a region surrounded by an outer wall line in the handwritten drawing shown in FIG. 2. In FIG. 7, the region surrounded by the outer wall line is hatched. For example, the element data output model 121 applies semantic segmentation to the handwritten drawing shown in FIG. 2 to identify a region surrounded by the outer wall line in the handwritten drawing. Then, the element data output model 121 outputs element data indicating the position of the region indicated by hatching in FIG. 7.

[0033] FIG. 8 is a diagram showing an example of an image obtained by identifying regions in which each room is drawn in the handwritten drawing shown in FIG. 2 using semantic segmentation. In FIG. 8, ten regions in which rooms are drawn are hatched differently from each other. For example, the element data output model 121 applies semantic segmentation to the handwritten drawing shown in FIG. 2 to identify regions in which rooms are drawn in the handwritten drawing. Then, the element data output model 121 outputs element data indicating the positions of the regions hatched in FIG. 8.

[0034] The element acquisition function 122 determines whether it is necessary to execute the process of classifying the elements indicated by the element data for outputting the detailed data once or more than once. When the element acquisition function 122 determines that it is sufficient to execute the process of classifying the elements indicated by the element data for outputting the detailed data once, the element acquisition function 122 acquires the element data related to the element and inputs it to the detailed data output model 123. On the other hand, when the element acquisition function 122 determines that it is necessary to execute the process of classifying the elements indicated by the element data for outputting the detailed data more than once, the element acquisition function 122 does not acquire the element data related to the element.

[0035] The detailed data output model 123 is, for example, a neural network. The detailed data output model 123 executes, for each element, a process of outputting detailed data indicating information about the element based on the element data. The information about the element mentioned here is, for example, data related to at least one of the specification, manufacture, assembly, and construction of the element. Also, the term "manufacture" means manufacturing parts, members, etc. that are elements in a factory. Also, the term "assembly" means combining a plurality of elements. Also, the term "construction" means, when the design object is a building, combining each element at a construction site to construct the building.

[0036] For example, the detailed data output model 123 executes a process of outputting detailed data including member data indicating information for identifying a member that realizes an element and direction data indicating the direction in which the member is attached to the design target. Note that the elements to which the process by the detailed data output model 123 is applied are determined in advance for each element.

[0037] The element acquisition function 124 determines whether it is necessary to execute the process of classifying the elements indicated by the element data for outputting the detailed data once or more than once. When the element acquisition function 124 determines that it is sufficient to execute the process of classifying the elements indicated by the element data for outputting the detailed data once, the element data related to the element is not acquired. On the other hand, when the element acquisition function 124 determines that it is necessary to execute the process of classifying the elements indicated by the element data for outputting the detailed data more than once, the element data related to the element is acquired and input to the member identification model 125.

[0038] The member identification model 125 is, for example, a neural network. The member identification model 125 outputs member data indicating information for identifying a member that realizes an element based on the element data. The member acquisition function 126 acquires the member data and inputs it to the direction identification model 127. The direction identification model 127 is, for example, a neural network. The direction identification model 127 outputs direction data indicating the direction in which the member is attached to the design target based on the member data. Note that the model combining the member identification model 125 and the direction identification model 127 may be regarded as the detailed data output model. Also, the elements to which the process by the member identification model 125 and the process by the direction identification model 127 are applied are determined in advance for each element.

[0039] Note that at least one of the element data output model 121, the detailed data output model 123, and the member identification model 125 preferably includes at least one of a FCN (Fully Convolutional Network), an Hour-glass type model, an encoder-decoder structure, an Efficient Net, a DeepLab, and a Residual Connection according to the characteristics of the process it executes.

[0040] The FCN is the first semantic segmentation model using deep learning, and an encoder-decoder structure including an encoder and a decoder is adopted. The encoder included in the FCN is, for example, one in which the fully connected layer and the pooling layer immediately before each fully connected layer are removed from a model such as ResNet used for image classification, and is used to learn features. The decoder included in the FCN applies deconvolution, the nearest neighbor algorithm, bilinear interpolation, etc. to restore the resolution reduced by the encoder, and performs pixel-level prediction from the features.

[0041] Similar to the above-described ResNet, the EfficientNet is a model in which unnecessary layers are removed from the model used for image classification and can be used as an encoder. In addition, since the prediction accuracy of the EfficientNet depends on the number of layers it includes, the number of filters used in each convolutional layer, and the resolution of the image input to it, these parameters that affect the prediction accuracy are uniformly determined based on the newly added parameter φ. As a result, the EfficientNet has the advantage that it can efficiently determine the required memory capacity, the amount of calculation, etc. according to the constraints.

[0042] DeepLab is a model that employs an encoder-decoder architecture, similar to FCN. DeepLab also uses a module called Atrous Spatial Pyramid Pooling (ASPP) before the decoder to efficiently acquire the context necessary for semantic segmentation. "Context" here refers to the information about surrounding pixels needed to predict a given pixel. Note that the class of a given pixel cannot be predicted based on information about a single pixel alone. ASPP also uses a dilated convolutional layer and a global pooling layer. This allows ASPP to acquire context from a wide range while avoiding the problem of increasing the size of the filter used in the convolutional layer, which can acquire context from a wide range but requires large memory capacity and computational effort, which can hinder learning.

[0043] Residual Connection is a structure in which the input to a certain layer in a model is added to the output of any layer after that layer. A layer whose output is added to the input of the previous layer by Residual Connection learns the residual with that input through training. This allows Residual Connection to train stably even in models with an extremely large number of layers, which was previously difficult to train. The ResidualNet mentioned above is a stack of these Residual Connections.

[0044] The post-processing function 13 performs post-processing on at least a portion of the contour of the region based on at least one of the element data and the detailed data, and outputs post-processed data indicating at least a portion of the contour of the region. The post-processing here refers to, for example, applying semantic segmentation to clarify and extract the contour, vertices, centerline, etc. of the identified region.

[0045] FIG. 9 is a diagram showing an example of an image obtained by applying post-processing to the image shown in FIG. 4. For example, the post-processing function 13 extracts the contours of each area surrounded by areas drawn with dot hatching or diagonal hatching in the image shown in FIG. 4. Next, the post-processing function 13 identifies each line segment that constitutes the center line of the area drawn with dot hatching or diagonal hatching in the image shown in FIG. 4 and is parallel to the vertical or horizontal direction in FIG. 9 based on the contours. Then, in a portion of the area drawn with dot hatching or diagonal hatching in the image shown in FIG. 4 where two or more line segments are identified, the post-processing function 13 deletes at least some of the line segments until only one line remains. By performing this processing, the post-processing function 13 generates and outputs the image shown in FIG. 9.

[0046] Note that the post-processing function 13 may perform such processing only on the area depicted with dot hatching in the image shown in FIG. 4 or only on the area depicted with diagonal hatching in the image shown in FIG. 4.

[0047] Fig. 10 is a diagram showing an example of an image obtained by applying post-processing to the image shown in Fig. 5. For example, the post-processing function 13 extracts a contour for each region depicted in the image shown in Fig. 5 and identifies the top left point and the bottom right point from among the points constituting the contour, thereby generating and outputting the image shown in Fig. 10. Note that the post-processing function 13 may also extract a contour for each region depicted in the image shown in Fig. 5 and identify the top right point and the bottom left point from among the points constituting the contour, thereby generating and outputting the image shown in Fig. 10.

[0048] FIG. 11 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 6. For example, the post-processing function 13 extracts the contour of the region depicted in the image shown in FIG. 6 by extracting a plurality of points that constitute the contour, and identifies the points characterizing the shape of the contour using the Douglas-Peucker method or the like. Then, the post-processing function 13 generates and outputs the image shown in FIG. 11 by adjusting the positions of the points characterizing the shape of the contour.

[0049] FIG. 12 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 7. For example, the post-processing function 13 extracts the contour of the region depicted in the image shown in FIG. 7 by extracting a plurality of points that constitute the contour, and identifies the points characterizing the shape of the contour using the Douglas-Peucker method or the like. Then, the post-processing function 13 generates and outputs the image shown in FIG. 12 by adjusting the positions of the points characterizing the shape of the contour.

[0050] FIG. 13 is a diagram showing an example of an image obtained by performing post-processing on the image shown in FIG. 8. For example, the post-processing function 13 extracts the contour of each region depicted in the image shown in FIG. 8 by extracting a plurality of points that constitute the contour, and identifies the points characterizing the shape of the contour using the Douglas-Peucker method or the like. Then, the post-processing function 13 generates and outputs the image shown in FIG. 13 by adjusting the positions of the points characterizing the shape of the contour.

[0051] Further, before executing the processes described with reference to FIG. 9 and the processes described with reference to FIG. 13, the post-processing function 13 may execute a process of aligning the coordinates indicating the position of the area depicted in FIG. 4 with the coordinates indicating the position of the area depicted in FIG. 8. For example, the post-processing function 13 further accurately specifies the position of at least one of the wall and the fixture by adjusting the position and length of at least one of the above-described line segments. Next, the post-processing function 13 specifies a closed area surrounded by a plurality of line segments. Then, the post-processing function 13 decomposes each closed area into a plurality of rectangles. Then, for each area including a plurality of rectangles, the post-processing function 13 executes a majority vote on the meaning such as the type and name of the object depicted by each rectangle, and determines that the area indicates the meaning such as the type and name of the object with the most rectangles depicted.

[0052] Note that the learning data acquisition function 11 may acquire learning data including data indicating a digitized drawing obtained by digitizing a handwritten drawing shown by the handwritten drawing data. FIG. 14 is a diagram showing an example of the digitized drawing included as an answer in the learning data according to the embodiment. For example, the learning data acquisition function 11 acquires learning data including data indicating the digitized drawing shown in FIG. 14. In such a case, the handwritten drawing data becomes the problem, and the data indicating the digitized drawing becomes the answer. Therefore, in this case, the machine learning model 120 executes supervised learning. On the other hand, when the machine learning model 120 includes the handwritten drawing data and does not include the data indicating the digitized drawing, the machine learning model 120 executes unsupervised learning.

[0053] Next, an example of the process executed by the machine learning execution program 10 according to the embodiment will be described with reference to FIG. 15. FIG. 15 is a flowchart showing an example of the process executed by the machine learning execution program according to the embodiment.

[0054] In step S11, the learning data acquisition function 11 acquires learning data including handwritten drawing data.

[0055] In step S12, the machine learning execution function 12 determines whether it is necessary to execute the process of classifying the elements indicated by the element data for outputting detailed data two or more times. If the machine learning execution function 12 determines that it is not necessary to execute the process of classifying the elements indicated by the element data for outputting detailed data two or more times, that is, if it is determined that the process needs to be executed only once (step S12: NO), the process proceeds to step S13. On the other hand, if the machine learning execution function 12 determines that it is necessary to execute the process of classifying the elements indicated by the element data for outputting detailed data twice (step S12: YES), the process proceeds to step S14.

[0056] In step S13, the machine learning execution function 12 inputs learning data into a machine learning model including a detailed data output model that executes a process of outputting detailed data including member data and direction data to update the machine learning model.

[0057] In step S14, the machine learning execution function 12 inputs learning data into a machine learning model including a detailed data output model including a member identification model that outputs member data based on element data and a direction identification model that outputs direction data based on member data to update the machine learning model.

[0058] Next, with reference to FIG. 16, an example of the process executed when the machine learning execution program corrects a handwritten drawing will be described. FIG. 16 is a diagram showing an example of the process executed by the machine learning execution program according to the embodiment when correcting a handwritten drawing.

[0059] The learning data acquisition function 11 acquires handwritten drawing data indicating a handwritten drawing in which at least one of a deletion element to be deleted from the handwritten drawing data and an addition element to be added to the handwritten drawing data is drawn.

[0060] The machine learning execution function 12 inputs at least one of the deletion element and the element data indicating the position of the area where the deletion element is drawn and the addition element and the element data indicating the position of the area where the addition element is drawn into the element data output model 121 to train the machine learning model 120. Then, based on the handwritten drawing data acquired by the learning data acquisition function 11, the element data output model 121 outputs at least one of the deletion element and the element data indicating the position of the area where the deletion element is drawn and the addition element and the element data indicating the position of the area where the addition element is drawn.

[0061] The post-processing function 13 executes post-processing based on at least one of the element data regarding the deletion element and the element data regarding the addition element, and outputs post-processed data indicating at least a part of the contour of the area. The post-processed data is data used when the handwritten drawing correction device 14 shown in FIG. 16 corrects the handwritten drawing based on at least one of the element data regarding the deletion element and the element data regarding the addition element and outputs corrected handwritten drawing data indicating the corrected handwritten drawing.

[0062] Next, an example of the data conversion program according to the embodiment will be described with reference to FIGS. 17 to 22. FIG. 17 is a diagram showing an example of the functional configuration of the data conversion program according to the embodiment. As shown in FIG. 17, the data conversion program 20 includes an inference data acquisition function 21, a detailed data output function 22, and a data conversion function 23.

[0063] At least a part of the functions of the machine learning execution program 10 is realized, for example, by a hardware processor such as a CPU reading and executing the machine learning execution program 10 stored in the storage device. Here, the storage device mentioned here is, for example, a hard disk drive, a solid state drive, a flash memory, or a ROM.

[0064] The inference data acquisition function 21 acquires, as inference data, inference handwritten drawing data indicating an inference handwritten drawing that is a drawing related to the design of the design object for inference and on which handwritten graphics are drawn. FIG. 18 is a diagram showing an example of a drawing that can be handled on a computer-aided design system. FIG. 19 is a diagram showing an example of a drawing obtained by handwritten correction of the drawing shown in FIG. 18. In the drawing shown in FIG. 19, handwritten corrections are added to the portion indicated by the frame L1 and the portion indicated by the frame L2. For example, the inference data acquisition function 21 acquires, as inference data, inference handwritten drawing data indicating the drawing shown in FIG. 19.

[0065] The detailed data output function 22 inputs the inference data into the above-described machine learning model 120 and causes the machine learning model 120 to output detailed data regarding the inference data.

[0066] The data conversion function 23 converts the inference handwritten drawing indicated by the inference handwritten drawing data based on the detailed data regarding the inference data and outputs a converted drawing. The data indicating the converted drawing is, for example, vector data including a tag indicating the type of an element and coordinates indicating the position of the element. FIG. 20 is a diagram showing an example of a converted drawing according to the embodiment. For example, the data conversion function 23 converts the drawing shown in FIG. 19 based on the detailed data regarding the inference handwritten drawing data indicating the drawing shown in FIG. 19, and causes the machine learning model 120 to output the converted drawing shown in FIG. 20.

[0067] FIG. 21 is a diagram showing an example of a tree structure of elements constituting the design object for inference according to the embodiment. For example, as shown in FIG. 21, the design object for inference has a tree structure including elements A, B, C, D, and E. Element A includes elements A1 and A2. Element C includes elements C1 and C2. Further, element C2 includes elements C21 and C22. Element E includes element E1. Further, element E1 includes elements E11 and E12.

[0068] In such a case, for example, element C becomes a deletion element, and element C, and elements C1 and C2 included in element C are deleted from the tree structure shown in FIG. 21. Similarly, elements C21 and C22 included in element C2 are deleted from the tree structure shown in FIG. 21. Also, in such a case, for example, element E1 becomes an addition element, and element E1, and elements E11 and E12 included in element E1 are added to the tree structure shown in FIG. 21.

[0069] Next, an example of the process executed by the data conversion program 20 according to the embodiment will be described with reference to FIG. 22. FIG. 22 is a flowchart showing an example of the process executed by the data conversion program according to the embodiment.

[0070] In step S21, the inference data acquisition function 21 acquires inference handwritten drawing data as inference data.

[0071] In step S22, the detailed data output function 22 inputs the inference data into a machine learning model including an element data output model and a detailed data output model to output detailed data regarding the inference data.

[0072] In step S23, the data conversion function 23 converts the inference handwritten drawing indicated by the inference handwritten drawing data into data based on the detailed data regarding the inference data, and outputs a converted drawing.

[0073] As described above, the machine learning execution program 10 and the data conversion program 20 according to the embodiment have been described. The machine learning execution program 10 inputs learning data including handwritten drawing data indicating a handwritten drawing into the above-described machine learning model 120 to update the machine learning model 120. Thereby, the machine learning execution program 10 can generate a machine learning model 120 that can accurately convert a handwritten drawing into a data format that can be handled on a computer-aided design system.

[0074] Also, when the machine learning execution program 10 only needs to execute the process of classifying the elements indicated by the element data once to output the detailed data, it inputs the learning data into the machine learning model 120 including the detailed data output model that executes the process of outputting the detailed data including the member data and the direction data. Thereby, the machine learning execution program 10 can quickly output the detailed data by comprehensively considering both the member data and the direction data at once. Also, such a process is effective for elements that require determining both the information identifying the member that realizes the element and the direction in which the member is attached to the design target. Examples of such elements include a kitchen set and a toilet.

[0075] Also, when the machine learning execution program 10 needs to execute the process of classifying the elements indicated by the element data two or more times to output the detailed data, it inputs the learning data into the machine learning model including the detailed data output model including the member identification model 125 and the direction identification model 127. Thereby, the machine learning execution program 10 can output accurate detailed data for members for which it is preferable to output the detailed data based on the direction in which the member is attached to the design target after narrowing down to a certain extent based on the information identifying the member that realizes the element. Examples of such elements include a right-opening door and a left-opening door.

[0076] Also, the machine learning execution program 10 executes post-processing based on at least one of the element data and the detailed data for at least a part of the contour of the region and outputs post-processed data indicating at least a part of the contour of the region. Thereby, the machine learning execution program 10 can clarify the contour, vertices, etc. of the region to improve the quality of the learning data and improve the learning accuracy of the machine learning model 120.

[0077] Further, the machine learning execution program 10 inputs into a machine learning model 120 including an element data output model 121 that outputs at least one of element data regarding deletion elements and element data regarding addition elements based on handwritten drawing data. Thereby, the machine learning execution program 10 can update the machine learning model 120 to a machine learning model that is detailed data regarding at least one of deletion elements and addition elements and can output detailed data that can be used for output of corrected handwritten drawing data.

[0078] Also, the data conversion program 20 inputs inference handwritten drawing data indicating an inference handwritten drawing as inference data into the machine learning model 120 that has been updated, and causes the machine learning model 120 to output detailed data regarding the inference data. Then, the data conversion program 20 converts the inference handwritten drawing indicated by the inference handwritten drawing data into data based on the detailed data regarding the inference data and outputs a converted drawing. Thereby, the data conversion program 20 can accurately convert a handwritten drawing into a data format that can be handled on a computer-aided design system.

[0079] Note that, in the above-described embodiment, the case where each function shown in FIG. 1 is realized by a hardware processor that reads and executes the machine learning execution program 10 has been described as an example, but the present invention is not limited to this. Also, in the above-described embodiment, the case where each function shown in FIG. 17 is realized by a hardware processor that reads and executes the data conversion program 20 has been described as an example, but the present invention is not limited to this.

[0080] At least a part of the functions shown in FIG. 1 and at least a part of the functions shown in FIG. 17 may be realized by hardware including circuitry such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit). Alternatively, at least a part of the functions shown in FIG. 1 and at least a part of the functions shown in FIG. 17 may be realized by the cooperation of software and hardware. Also, these hardware components may be integrated into one or divided into multiple components.

[0081] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration of the embodiments of the present invention is not limited to the above-described embodiments, and at least one of various combinations, modifications, substitutions, and design changes may be added to the above-described embodiments without departing from the gist of the present invention.

Explanation of Reference Numerals

[0082] 10… Machine learning execution program, 11… Learning data acquisition function, 12… Machine learning execution function, 13… Post-processing function, 20… Data conversion program, 21… Inference data acquisition function, 22… Detailed data output function, 23… Data conversion function

Claims

1. A computer, a learning data acquisition function for acquiring learning data including handwritten drawing data that is a drawing related to a design target and shows a handwritten drawing with handwritten figures drawn thereon, an element acquisition function for determining the number of times of processing for classifying elements indicated by element data in order to output detailed data that is information indicating elements constituting the design target, an element data output model that executes, for each element, a process of specifying a region in which the element is drawn in the handwritten drawing indicated by the handwritten drawing data and outputting element data indicating the type of the element and the position of the region, and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information related to the element based on the element data, the detailed data output model having a model structure based on the determination result of the number of times of the process, and a machine learning execution function for inputting the learning data into the machine learning model including the above to update the machine learning model, A machine learning execution program for realizing the above.

2. When the machine learning execution function only needs to execute the process of classifying the elements indicated by the element data once to output the detailed data, Input the learning data into the machine learning model including the detailed data output model that executes a process of outputting the detailed data including member data indicating information for specifying a member that realizes the element and direction data indicating a direction in which the member is attached to the design target. The machine learning execution program according to Claim 1.

3. When the machine learning execution function needs to execute the process of classifying the elements indicated by the element data two or more times to output the detailed data, Input the learning data into the machine learning model including the detailed data output model including a member specification model that outputs member data indicating information for specifying a member that realizes the element based on the element data and a direction specification model that outputs direction data indicating a direction in which the member is attached to the design target based on the member data. The machine learning execution program according to Claim 1.

4. The computer is further caused to realize a post-processing function for extracting at least a part of the contour of the region based on at least one of the element data and the detailed data and outputting post-processed data indicating at least a part of the contour of the region. The machine learning execution program according to claim 2 or claim 3.

5. The machine learning execution function inputs the learning data into the machine learning model including the element data output model that outputs at least one of the deletion elements to be deleted from the handwritten drawing indicated by the handwritten drawing data and the element data indicating the position of the area where the deletion elements are drawn, and the addition elements to be added to the handwritten drawing indicated by the handwritten drawing data and the element data indicating the position of the area where the addition elements are drawn, based on the handwritten drawing data. The machine learning execution program according to any one of claims 1 to 4.

6. On a computer, an inference data acquisition function that acquires, as inference data, inference handwritten drawing data indicating an inference handwritten drawing that is a drawing related to a design object for inference and has handwritten figures drawn thereon; an element acquisition function that determines the number of times of processing for classifying the elements indicated by the element data in order to output detailed data that is information indicating the elements constituting the design object for inference; an element data output model that executes, for each element, a process of specifying the area where the element is drawn in a handwritten drawing that is a drawing related to a design object and has handwritten figures drawn thereon, and outputting element data indicating the type of the element and the position of the area, and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information about the element based on the element data, the detailed data output model having a model structure based on the determination result of the number of times of the process, and inputs the inference data into the machine learning model including the element data output model and the detailed data output model, and causes the machine learning model to output the detailed data related to the inference data; a data conversion function that converts the inference handwritten drawing indicated by the inference handwritten drawing data based on the detailed data related to the inference data and outputs a converted drawing; A data conversion program for realizing the above.

7. a learning data acquisition unit that acquires learning data including handwritten drawing data indicating a handwritten drawing that is a drawing related to a design object and has handwritten figures drawn thereon; an element acquisition function that determines the number of times of processing for classifying the elements indicated by the element data in order to output detailed data that is information indicating the elements constituting the design object; In the handwritten drawing indicated by the handwritten drawing data, identify the area where the element is drawn, and execute for each element a process of outputting information identifying the type of the element and element data indicating the position of the area, an element data output model; and a detailed data output model that executes for each element a process of outputting detailed data indicating information about the element based on the element data, the detailed data output model having a model structure based on a determination result of the number of times of the process; and a machine learning execution unit that inputs the learning data into the machine learning model including the above and updates the machine learning model. A machine learning execution device comprising the above. [

8. ] An inference data acquisition unit that acquires, as inference data, inference handwritten drawing data indicating an inference handwritten drawing that is a drawing related to a design object for inference and in which a handwritten figure is drawn; An element acquisition unit that determines the number of times of a process of classifying elements indicated by element data in order to output detailed data that is information indicating elements constituting the design object for inference; In the handwritten drawing that is a drawing related to the design object and in which a handwritten figure is drawn, identify the area where the element is drawn, and execute for each element a process of outputting information identifying the type of the element and element data indicating the position of the area, an element data output model; and a detailed data output model that executes for each element a process of outputting detailed data indicating information about the element based on the element data, the detailed data output model having a model structure based on a determination result of the number of times of the process; and a detailed data output unit that inputs the inference data into the machine learning model including the above and causes the machine learning model to output the detailed data related to the inference data. A data conversion unit that converts the inference handwritten drawing indicated by the inference handwritten drawing data based on the detailed data related to the inference data and outputs a converted drawing. A data conversion device comprising the above. [

9. ] The input unit of the computer acquires learning data including handwritten drawing data indicating a handwritten drawing that is a drawing related to the design object and in which a handwritten figure is drawn, determines the number of times of a process of classifying elements indicated by element data in order to output detailed data that is information indicating elements constituting the design object, The arithmetic unit of the computer executes, for each element, a process of identifying a region in the handwritten drawing indicated by the handwritten drawing data where the element is drawn, and outputting information identifying the type of the element and element data indicating the position of the region, and an element data output model; and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information about the element based on the element data, the detailed data output model having a model structure based on a determination result of the number of times of the process, and inputs the learning data into a machine learning model stored in a storage unit of the computer to update the machine learning model. Machine learning execution method.

10. An input unit of a computer acquires, as inference data, inference handwritten drawing data indicating an inference handwritten drawing that is a drawing related to a design object for inference and in which handwritten figures are drawn. An element acquisition unit of the computer determines the number of times of a process of classifying elements indicated by element data in order to output detailed data that is information indicating elements constituting the design object for inference. The arithmetic unit of the computer executes, for each element, a process of identifying a region in the handwritten drawing that is a drawing related to the design object and in which handwritten figures are drawn, where the element is drawn, and outputting information identifying the type of the element and element data indicating the position of the region, and an element data output model; and a detailed data output model that executes, for each element, a process of outputting detailed data indicating information about the element based on the element data, the detailed data output model having a model structure based on a determination result of the number of times of the process, and inputs the inference data into a machine learning model stored in a storage unit of the computer to cause the machine learning model to output the detailed data related to the inference data. An output unit of the computer converts the inference handwritten drawing indicated by the inference handwritten drawing data into data based on the detailed data related to the inference data and outputs a data-converted drawing. Data conversion method.

Images