Model training method, labeling method, electronic equipment, storage medium and product

Abstract

The invention provides a model training method, a labeling method, electronic equipment, a storage medium and a product. The model training method comprises the following steps: determining labeled drawing data; displaying drawing information, first labeled object feature information, first labeled content, second labeled object feature information, second lead information, second position information and candidate position information in the labeled drawing data are input into a neural network model, and predicted position information and predicted lead information are obtained; calculating a target loss value based on the first position information in the labeled drawing data, the first lead information in the labeled drawing data, the predicted position information and the predicted lead information; and adjusting parameters of the neural network model based on the target loss value to obtain a trained neural network model.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as drawing display, and in particular to a model training method, a labeling method, an electronic device, a storage medium and a product. Background Art

[0002] In the fields of engineering design, architectural planning, mechanical manufacturing, etc., there are strict requirements on the accuracy of drawings. Therefore, the annotation information in the drawings naturally becomes the key link in accurately conveying the drawing information.

[0003] In the existing related technologies, a common practice is for relevant professionals to manually add relevant annotation content where annotation is required in the drawings.

[0004] However, in actual scenarios, drawings are generally more complex, and the number of objects that need to be annotated is large, which requires a lot of time and effort. This situation reflects that under the existing technical environment, there are significant defects in the annotation process and the annotation efficiency is low. Summary of the invention

[0005] The present disclosure provides a model training method, a labeling method, an electronic device, a storage medium and a product.

[0006] According to one aspect of the present disclosure, a model training method is provided, comprising: Determine annotated drawing data, wherein the annotated drawing data includes display drawing information, feature information of a first annotated object in the display drawing, first annotation content and first lead information corresponding to the first annotated object, first position information of the first annotation content, feature information of a second annotated object in the display drawing, second lead information corresponding to the second annotated object, second position information of a second annotation content corresponding to the second annotated object, and candidate position information; Inputting the display image information, the first annotated object feature information, the first annotated content, the second annotated object feature information, the second lead information, the second position information, and the candidate position information into a neural network model, and predicting the position information of the first annotated content and the lead information corresponding to the first annotated object through the neural network model to obtain predicted position information and predicted lead information; calculating a target loss value based on the first position information, the first lead information, the predicted position information, and the predicted lead information; and The parameters of the neural network model are adjusted based on the target loss value to obtain a trained neural network model, wherein the trained neural network model can predict the target position information and target lead information of the annotation content corresponding to the annotated object in the drawing to be annotated.

[0007] According to at least one embodiment of the model training method of the present disclosure, based on the first position information, the first lead information, the predicted position information and the predicted lead information, calculating the target loss value includes: determining a first loss value according to a difference between the first position information and the predicted position information; determining a second loss value according to a difference between the first lead information and the predicted lead information; and The target loss value is determined at least according to the sum of the first loss value and the second loss value.

[0008] According to the model training method of at least one embodiment of the present disclosure, the first loss value is obtained through a first cross entropy loss function, and the second loss value is obtained through a second cross entropy loss function.

[0009] According to at least one embodiment of the present disclosure, the model training method further includes, before determining the target loss value at least according to the sum of the first loss value and the second loss value: Determine one or more of the number of lead intersections, the angle difference, and the length in the display diagram according to the predicted lead information, wherein the number of lead intersections is the number of times the predicted lead corresponding to the predicted lead information and the second lead corresponding to the second lead information cross each other, the angle difference is the difference between the angle between the predicted lead and the reference line and the target expected angle, the target expected angle is the expected angle corresponding to the angle interval in which the angle is located, and the length in the display diagram is the length of the predicted lead in the display diagram; and Calculating a third loss value according to one or more of the number of lead intersections, the angle difference, and the length in the display diagram; Determining the target loss value at least according to the sum of the first loss value and the second loss value includes: The sum of the first loss value, the second loss value and the third loss value is taken as the target loss value.

[0010] According to the model training method of at least one embodiment of the present disclosure, calculating the third loss value according to one or more of the number of intersections of the lead lines, the angle difference, and the length in the display diagram includes: determining a first score according to the number of intersections of the lead lines, wherein the first score is positively correlated with the number of intersections of the lead lines; determining a second score based on the angular difference, the second score being positively correlated with the angular difference; Determine a third score according to the length of the display image, wherein the third score is positively correlated with the length of the display image; and The first score, the second score and the third score are weightedly summed to obtain the third loss value.

[0011] According to the model training method of at least one embodiment of the present disclosure, the display image information includes the display image boundary and / or the display image centroid point.

[0012] According to the model training method of at least one embodiment of the present disclosure, the first labeled object feature information and the second labeled object feature information respectively include one or more of the labeled object type, the labeled object centroid point, and the labeled object boundary.

[0013] According to the model training method of at least one embodiment of the present disclosure, the first lead information and the second lead information respectively include multiple ones of the lead connection points on the annotated object, the lead connection points at the positions where the annotation content corresponding to the annotated object is located, the lead turning points and the lead types.

[0014] According to another aspect of the present disclosure, there is provided a marking method, comprising: Determine the drawing data to be annotated, wherein the drawing data to be annotated includes at least the display drawing information, the feature information of the object to be annotated in the display drawing, the content to be annotated corresponding to the object to be annotated, and the candidate position information; Inputting the drawing data to be annotated into a trained neural network model, predicting the position information of the content to be annotated and the lead information corresponding to the object to be annotated by the neural network model, and obtaining target position information and target lead information, wherein the neural network model is obtained by the model training method described in any one of the above embodiments; Displaying the content to be annotated at a target position in the drawing to be annotated, wherein the target position is obtained through the target position information; and Based on the target lead information, a target lead is generated to connect the displayed content to be annotated and the object to be annotated corresponding to the displayed content to be annotated.

[0015] According to the marking method of at least one embodiment of the present disclosure, the target lead information includes a first point, a second point and a turning point, the first point is the connection point of the target lead on the object to be marked, the second point is the connection point of the target lead at the target position, and the target lead can reach the turning point from the first point along a first direction, and reach the second point along a second direction after reaching the turning point.

[0016] According to at least one embodiment of the present disclosure, the marking method inputs the drawing data to be marked into a trained neural network model, including: In response to the drawing data to be annotated also including annotated information, inputting the display drawing information, the feature information of the object to be annotated, the content to be annotated, the annotated information and the candidate position information into the trained neural network model, wherein the annotated information includes the feature information of the annotated object, the known lead information corresponding to the annotated object and the known position information of the annotated content corresponding to the annotated object; and In response to the fact that the to-be-annotated drawing data does not include the annotated information, the display image information, the feature information of the to-be-annotated object, the to-be-annotated content, the preset empty list and the candidate position information are input into the trained neural network model.

[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the model training method or labeling method of any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the model training method or labeling method of any embodiment of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the model training method or labeling method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0021] Figure 1 It is a flowchart of a model training method according to an embodiment of the present invention.

[0022] Figure 2 It is a schematic diagram of a process of calculating a target loss value according to an embodiment of the present disclosure.

[0023] Figure 3 It is a schematic diagram of a process of calculating a target loss value according to another embodiment of the present disclosure.

[0024] Figure 4 It is a schematic diagram of the process of calculating the third loss value according to an embodiment of the present disclosure.

[0025] Figure 5It is a visual example diagram of the model training process of one embodiment of the present disclosure.

[0026] Figure 6 It is a flowchart of a marking method according to an embodiment of the present disclosure.

[0027] Figure 7 It is a schematic block diagram of the structure of a model training device according to an embodiment of the present invention.

[0028] Figure 8 It is a schematic block diagram of the structure of a labeling device according to an embodiment of the present disclosure.

[0029] Fig. 9 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0031] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] Drawings are widely used in all walks of life, and the setting of annotation content in drawings is a key link to ensure that the information in the drawings is accurately conveyed to readers. In the decoration industry, the annotation content in the floor plan is often used to understand the layout, structure and size of the house, as well as the decoration materials, furniture, construction details and other conditions in various locations of the house. If the method of manually adding annotation content to the places where annotation is required in the drawings is adopted, then when the number of annotated objects included in the drawings is large, the relevant personnel need to invest a lot of time and energy to think about how to set the location information of the annotation content and how to set the lead lines used to connect the annotation content and the corresponding annotated objects, which leads to low annotation efficiency.

[0033] To this end, the present disclosure proposes a model training method and a labeling method.

[0034] The model training method disclosed in the present invention can be used by a first electronic device to automatically train a neural network model when receiving drawing data, thereby obtaining target position information in the drawing that can better predict the annotation content corresponding to the annotated object and the target lead information for connecting the annotated object and the annotation content.

[0035] The model training method disclosed in the present invention uses annotated drawing data to train a neural network model, so that the neural network model can learn the accurate position of the first annotation content corresponding to the first annotated object on the drawing and how to correctly connect the first annotation content with the corresponding first annotated object, and then the trained neural network model can automatically and accurately predict the target position information and target lead information of the annotation content corresponding to the annotated object in the drawing to be annotated, avoiding manual thinking about how to set the position information and lead lines of the annotation content, which is conducive to improving the annotation efficiency.

[0036] In the present disclosure, the first electronic device includes but is not limited to a server, a mobile phone, a tablet computer, a laptop computer, a personal computer, a wearable device, an ATM, etc.

[0037] The labeling method disclosed in the present invention can be used by an electronic device to determine, when receiving drawing data to be labeled, the target position information of the labeled content corresponding to the labeled object in the drawing and the target lead information for connecting the labeled object and the labeled content through a pre-trained neural network model, and then complete the labeling based on the target position information and the target lead information.

[0038] In the present disclosure, the second electronic device includes but is not limited to a server, a mobile phone, a tablet computer, a laptop computer, a personal computer, a wearable device, an ATM, etc. The first electronic device and the second electronic device may be the same device or different devices, which is not limited here.

[0039] For the convenience of description and to make the technical solutions of the specific implementation methods of the present disclosure easier to understand, before describing the model training method implemented in the present disclosure, the technical terms involved in the specific implementation methods of the present disclosure are explained as follows: Drawings: important documents used to convey design intent, record technical information and guide construction or production in the fields of engineering technology, architectural design, mechanical manufacturing, etc.

[0040] Annotation: refers to the process of establishing a connection between a specific object in a drawing (i.e. the annotated object) and the annotated content.

[0041] Annotated drawings: drawings in which all annotated objects have been annotated.

[0042] Drawings to be annotated: drawings with at least one annotated object that has not been annotated yet.

[0043] Callouts: Graphic elements and / or text that provide additional information or explanation.

[0044] Leader: A line used to connect the annotated object and its corresponding annotation content.

[0045] Point type annotated object: that is, the annotated object is a point in the drawing.

[0046] Line type annotated object: that is, the annotated object is a line in the drawing.

[0047] The annotated object is a surface type: that is, the annotated object is a surface in the drawing.

[0048] Figure 1 FIG. 1 is a schematic diagram showing the overall process of a model training method M100 according to an embodiment of the present disclosure. Figure 1 The method shown includes steps S110 to S140. The method may be executed by a first electronic device such as a server.

[0049] Specifically, Figure 1 The methods shown include: S110, determining annotated drawing data, where the annotated drawing data includes display drawing information, feature information of a first annotated object in the display drawing, first annotation content and first lead information corresponding to the first annotated object, first position information of the first annotation content, feature information of a second annotated object in the display drawing, second lead information corresponding to the second annotated object, second position information of a second annotation content corresponding to the second annotated object, and candidate position information; The drawings may include presentation diagrams and Figure 4 The display diagram may be located in the center of the drawing, and may include multiple annotated objects. Any one of the multiple annotated objects may be used as the first annotated object, and the annotated objects other than the first annotated object may be used as the second annotated objects. The characteristic information of the first annotated object is some characteristic information of the first annotated object itself, and the characteristic information of the second annotated object is some characteristic information of the second annotated object itself. The first annotated content and the second annotated content may be displayed in the blank area of ​​the drawing, that is, the first position information and the second position information may be the positions in the blank area of ​​the drawing, respectively. The first position information can represent the position of the first annotated object in the display diagram. The second position information can represent the position of the second annotated object in the display diagram. The first lead information is the information of the first lead, and the first lead is used to connect the first annotated object and the first annotated content corresponding to the first annotated object, and the same is true for the second lead.

[0050] In one example, the display diagram is a floor plan, and multiple annotated objects include master bedroom sockets, living room sockets, second bedroom windows, entrance doors, etc. The annotated content corresponding to the annotated objects may be "window sill stone", "five-hole socket", "door stone", etc. In another example, multiple candidate boxes are set in the blank area of ​​the drawing, and a candidate position information may represent the position of a candidate box in the drawing. A first annotated content may be directly displayed in a candidate box or split and displayed in multiple candidate boxes. The position of one or more candidate boxes occupied by a first annotated content is the first position information of the first annotated content, and the same is true for the second annotated content and the second position information.

[0051] Exemplarily, when it is necessary to train a neural network model, multiple drawing files with annotation contents corresponding to each annotated object annotated in advance can be input into the first electronic device, the drawing file can be a JSON data file, and the feature information of the first annotated object in the display diagram of the drawing, the first annotation content and first lead information corresponding to the first annotated object, the first position information of the first annotation content, the feature information of the second annotated object in the display diagram, the second lead information corresponding to the second annotated object, the second position information of the second annotation content corresponding to the second annotated object, and the candidate position information can be saved in the above-mentioned JSON data file in the form of a string or a digital coordinate. Then, after receiving the drawing file, the first electronic device can determine the annotated drawing data through the algorithm for analyzing geometric objects in the relevant technology and / or the algorithm for parsing the drawing file.

[0052] S120, input the display image information, the feature information of the first annotated object, the first annotated content, the feature information of the second annotated object, the second lead information, the second position information and the candidate position information into the neural network model, and predict the position information of the first annotated content and the lead information corresponding to the first annotated object through the neural network model to obtain the predicted position information and the predicted lead information.

[0053] Among them, the neural network model of the present disclosure can be various existing neural network models, such as a neural network model based on deep learning, a shallow neural network model, etc., all of which fall within the protection scope of the present disclosure.

[0054] Preferably, the neural network model disclosed in the present invention is a neural network model based on deep learning. Considering that the trained neural network model is expected to be able to detect key points (i.e., the location information of the first annotation content and the lead information corresponding to the first annotated object), the neural network model is preferably an existing Detectron2 model or HRNet model.

[0055] The predicted position information can represent the position of the first annotated content predicted by the neural network model in the display image. The predicted lead information can represent the lead information corresponding to the first annotated object predicted by the neural network model.

[0056] S130, calculating a target loss value based on the first position information, the first lead information, the predicted position information, and the predicted lead information; For example, the target loss value can characterize the difference between the first position information and the predicted position information, and the difference between the first lead information and the predicted lead information. By calculating the target loss value, data support is provided for the neural network model parameter adjustment, which is conducive to training the neural network model in a more accurate direction, thereby saving the training time of the neural network model and improving the training efficiency.

[0057] S140. Adjust the parameters of the neural network model based on the target loss value to obtain a trained neural network model, wherein the trained neural network model can predict the target position information and target lead information of the annotation content corresponding to the annotated object in the drawing to be annotated.

[0058] Optionally, the training process of the neural network model includes m training rounds, and each training round uses multiple annotated drawing data. For each annotated drawing data, the first electronic device executes the above steps 120 and 130 to obtain the predicted position information and predicted lead information of the annotated drawing data. In step 140, the first electronic device adjusts the parameters of the neural network model according to the target loss value. It should be noted that since the predicted lead information is related to the predicted position information, for a certain annotated drawing data, the predicted position information corresponding to the annotated drawing data in different training rounds may be different, so the predicted lead information corresponding to the same annotated drawing data may also change in different training rounds. In this way, the predicted lead information changes dynamically with the parameter adjustment of the neural network model. As the neural network model improves its ability to predict position information, the predicted lead information corresponding to the annotated drawing data will be more accurate, which helps to improve the trained neural network model's ability to predict position information.

[0059] Exemplarily, if the target loss value converges, the training of the neural network model is completed; if the target loss value does not converge, a new round of training is performed on the neural network model. The labeled drawing data used in different training rounds can be the same or different. Optionally, the probability of occurrence of the labeled drawing data conforms to a uniform distribution. For example, the labeled drawing data used in different training rounds can be randomly extracted from the training sample set. The method for extracting the labeled drawing data is determined according to actual needs and is not limited here.

[0060] The model training method of the disclosed embodiment uses annotated drawing data to train a neural network model, so that the neural network model can learn the exact position of the first annotation content corresponding to the first annotated object on the drawing and how to correctly connect the first annotation content with the corresponding first annotated object, and then the trained neural network model can automatically and accurately predict the target position information and target lead information of the annotation content corresponding to the annotated object in the drawing to be annotated, avoiding manual thinking about how to set the position information and lead lines of the annotation content, which is conducive to improving the annotation efficiency.

[0061] In some embodiments of the present disclosure, the display image information includes one or more of a display image boundary and a display image centroid point.

[0062] In this way, the display diagram information can help the neural network model better understand the spatial structure of the display diagram, which is very helpful for locating the position information of the annotation content and determining the length of the lead line, which is conducive to improving the accuracy of the annotation. For example, the display diagram boundary can represent the position, shape and size of the display diagram in the drawing.

[0063] In some embodiments of the present disclosure, the first labeled object feature information and the second labeled object feature information respectively include one or more of a labeled object type, a labeled object centroid point, and a labeled object boundary.

[0064] Thus, the first annotated object feature information and the second annotated object feature information help the neural network model to identify and distinguish different annotated objects, especially in complex drawings, where such information is essential for correct annotation. Exemplarily, the annotated object type may include one or more of a point type, a line type, and a surface type. The annotated object boundary may characterize the position, shape, and size of the corresponding annotated object in the drawing.

[0065] In some embodiments of the present disclosure, the first lead information and the second lead information respectively include a plurality of the connection points of the lead on the annotated object, the connection points of the lead at the position of the annotation content corresponding to the annotated object, the lead turning points, and the lead type. The position of the annotation content corresponding to the annotated object can be obtained through the position information of the annotation content corresponding to the annotated object.

[0066] In this way, the first lead information and the second lead information help the neural network model to learn the generation rules of the lead more accurately, so that the lead information generated by the trained neural network model is more in line with the design specifications, which is conducive to enhancing the professionalism and readability of the drawings. Exemplarily, the two ends of the first lead are respectively connected to the first annotated object and the first position information, and the turning point of the first lead is located between the two endpoints of the first lead, which is used to characterize the position where the direction of the first lead changes, and the same is true for the second lead. The lead type may include one or more of a straight line, a broken line, and a curve.

[0067] Regarding step S130, in some embodiments of the present disclosure, it may include the following steps: Figure 2 Steps S131 to S133 are shown.

[0068] S131. Determine a first loss value according to a difference between the first position information and the predicted position information.

[0069] Exemplarily, the first loss value can be obtained by a first cross entropy loss function. For example, the first position information and the predicted position information can be substituted into the first cross entropy loss function to calculate the first loss value. In this way, the difference between the first position information and the predicted position information can be effectively constant through cross entropy, thereby helping to improve the learning ability of the neural network model for different annotation contents and annotated objects, thereby improving the prediction accuracy. In other embodiments, the first loss value can also be calculated by other convex functions, which is not limited in this disclosure.

[0070] S132: Determine a second loss value according to a difference between the first lead information and the predicted lead information.

[0071] Exemplarily, the second loss value can be obtained by a second cross entropy loss function. For example, the first lead information and the predicted lead information can be substituted into the second cross entropy loss function to calculate the second loss value. In this way, the difference between the first lead information and the predicted lead information can be effectively constant through cross entropy, thereby helping to improve the learning ability of the neural network model for different annotation contents and annotated objects, thereby improving the prediction accuracy. In other embodiments, the second loss value can also be calculated by other convex functions, which is not limited in this disclosure.

[0072] S133. Determine a target loss value based at least on the sum of the first loss value and the second loss value.

[0073] For example, the sum of the first loss value and the second loss value can be directly used as the target loss value, so as to simply and quickly determine the target loss value, which is conducive to improving the efficiency of model training; other constraints can also be added on the basis of the sum of the first loss value and the second loss value to obtain the target loss value, so as to meet personalized needs and make the trained neural network model more in line with usage needs.

[0074] The model training method of the above-mentioned embodiment calculates the first loss value corresponding to the position difference and the second loss value corresponding to the lead difference, respectively, and determines the target loss value based on at least the sum of the first loss value and the second loss value, so that the subsequent adjustment of the parameters of the neural network model can optimize the predicted position information and the predicted lead information, avoiding the model bias problem that may be caused by a single loss value, and improving the overall performance of the model.

[0075] In some embodiments of the present disclosure, before step S133, the model training method may further include: Figure 3 Step S134 and step S135 are shown, and accordingly, step S133 may include step S1331.

[0076] S134. Determine one or more of the number of lead intersections, angle difference and length in the display diagram based on the predicted lead information, wherein the number of lead intersections is the number of times a predicted lead corresponding to the predicted lead information crosses with a second lead corresponding to the second lead information, the angle difference is the difference between the angle between the predicted lead and the reference line relative to the target expected angle, the target expected angle is the expected angle corresponding to the angle interval in which the angle is located, and the length in the display diagram is the length of the predicted lead in the display diagram.

[0077] Exemplarily, the first annotated object may correspond to a predicted leader information, one predicted leader information corresponds to a predicted leader, the predicted leader is used to connect the predicted position information and the first annotated object, one second leader information corresponds to a second leader, and then the number of times the predicted leader crosses with all the second leaders in the annotated drawing can be determined by the algorithm for analyzing geometric objects in the relevant technology. The angle between the predicted leader and the reference line and the length of the predicted leader in the display drawing can also be obtained by the algorithm for analyzing geometric objects in the relevant technology. The reference line can be a horizontal line or a vertical line. In the case where the predicted leader is a broken line, the angle between the predicted leader and the reference line can be understood as the angle between the line connecting the turning point of the predicted leader and the connection point on the first annotated object and the reference line.

[0078] S135. Calculate a third loss value according to one or more of the number of lead intersections, the angle difference, and the length in the display diagram.

[0079] Exemplarily, the third loss value can be calculated based only on the number of lead intersections, angle difference, or length in the display diagram, or based on any two of the number of lead intersections, angle difference, and length in the display diagram, or based on the number of lead intersections, angle difference, and length in the display diagram, without limitation herein. The third loss value can be understood as a geometric constraint loss, and by calculating the third loss value, the target loss value can be determined more accurately, thereby facilitating the neural network model to adjust parameters according to geometric constraints, so that the trained neural network model can output position information and lead information that meet the geometric constraints.

[0080] S1331. Take the sum of the first loss value, the second loss value and the third loss value as the target loss value.

[0081] In the model training method of the above-mentioned embodiment, the target loss value introduces a third loss value on the basis of the first loss value and the second loss value. Since the third loss value takes into account one or more factors of the number of lead intersections, angle difference and length in the display diagram, when the parameters of the neural network model are subsequently adjusted based on the target loss value, one or more of the number of lead intersections, angle difference and length in the display diagram can also be optimized, thereby improving the overall performance of the neural network model, improving the neural network model's prediction ability for leads with small number of lead intersections, small angle difference and short length in the display diagram, and improving the readability of the drawing.

[0082] Regarding step S135, in some embodiments of the present disclosure, it may include the following steps: Figure 4 Steps S1351 to S1354 are shown.

[0083] S1351. Determine a first score according to the number of lead intersections, where the first score is positively correlated with the number of lead intersections.

[0084] The number of lead line intersections can be directly used as the first score, and the more the number of lead line intersections, the larger the first score, and the fewer the number of lead line intersections, the smaller the first score, thereby improving the efficiency of determining the first score.

[0085] S1352. Determine a second score according to the angle difference, where the second score is positively correlated with the angle difference.

[0086] For example, the angle is [0 ,90 ) The expected angle corresponding to the interval is 45 , the angle is (-90 ,0 ] The expected angle corresponding to the interval is -45 , the angle is [90 ,180 ) The expected angle corresponding to the interval is 135 , the angle is (-180 ,90 ] The expected angle corresponding to the interval is -135 ; For the predicted lead, the corresponding angle difference can be calculated by the following formula 1.

[0087] Formula 1.

[0088] In formula 1, is the angle difference value corresponding to a predicted lead line, and x is the angle between the predicted lead line and the reference line. Furthermore, the angle difference value can be directly used as the second score.

[0089] S1353. Determine a third score according to the length of the display image, where the third score is positively correlated with the length of the display image.

[0090] The length in the display diagram can be directly used as the third score, and the longer the length in the display diagram is, the larger the third score is, and the shorter the length in the display diagram is, the smaller the third score is, thereby improving the efficiency of determining the third score.

[0091] S1354. Perform a weighted sum on the first score, the second score, and the third score to obtain a third loss value.

[0092] The weight values ​​corresponding to the first score, the second score and the third score can be set according to experience, and the present disclosure does not limit this. Since the first score, the second score and the third score are all as small as possible, the parameters of the neural network can be adjusted in the direction of reducing the first score, the second score and the third score.

[0093] It is worth noting that in other embodiments, only any one of the first score, the second score and the third score can be used as the third loss value, or any one of the first score, the second score and the third score can be used as the sum of any two of the third loss values ​​as the third loss value, and the present disclosure does not limit this.

[0094] The model training method of the above-mentioned embodiment calculates the third loss value based on the three factors of the number of lead intersections, the angle difference and the length in the display diagram. Since the first score is positively correlated with the number of lead intersections, the second score is positively correlated with the angle difference and the third score is positively correlated with the length in the display diagram, the number of lead intersections, the angle difference and the length in the display diagram can be uniformly represented by the size of the third loss value, thereby providing data support for the parameter adjustment of the neural network model and improving accuracy and convenience.

[0095] Please combine Figure 5In one example, the model training method may include the following steps: determining annotated drawing data, the annotated drawing data including display drawing information, feature information of a first annotated object in the display drawing, first annotation content and first lead information corresponding to the first annotated object, first position information of the first annotation content, feature information of a second annotated object in the display drawing, second lead information corresponding to the second annotated object, second position information of the second annotation content corresponding to the second annotated object, and candidate position information; inputting the display drawing information, feature information of the first annotated object, first annotation content, feature information of the second annotated object, second lead information, second position information, and candidate position information into a neural network model, and performing a neural network model on the position information of the first annotation content and the first lead information. Predict the lead information corresponding to the marked object to obtain predicted position information and predicted lead information; determine a first loss value based on the difference between the first position information and the predicted position information; determine a second loss value based on the difference between the first lead information and the predicted lead information; determine the number of lead intersections, angle difference and length in the display diagram based on the predicted lead information; determine a first score based on the number of lead intersections; determine a second score based on the angle difference; determine a third score based on the length in the display diagram; perform weighted summation of the first score, the second score and the third score to obtain a third loss value; use the sum of the first loss value, the second loss value and the third loss value as a target loss value; and adjust the parameters of the neural network model based on the target loss value to obtain a trained neural network model.

[0096] Figure 6 FIG. 2 shows a schematic diagram of the overall process of a labeling method M200 according to an embodiment of the present disclosure. Figure 6 The method shown includes steps S210 to S240. The method can be executed by a second electronic device such as a server, a mobile phone, or a computer.

[0097] Specifically, Figure 6 The methods shown include: S210, determining drawing data to be annotated, wherein the drawing data to be annotated includes at least display drawing information, feature information of the object to be annotated in the display drawing, content to be annotated corresponding to the object to be annotated, and candidate position information; The contents related to the feature information of the object to be labeled can refer to the description of the first labeled object mentioned above, and for the sake of brevity, they will not be repeated here.

[0098] S220, inputting the drawing data to be annotated into the trained neural network model, predicting the position information of the content to be annotated and the lead information corresponding to the object to be annotated by the neural network model, and obtaining the target position information and the target lead information, wherein the neural network model is obtained by the model training method of the above-mentioned embodiment; S230, displaying the content to be annotated at a target position in the drawing to be annotated, wherein the target position is obtained through target position information; S240 . Generate a target lead line connecting the displayed content to be annotated and the object to be annotated corresponding to the displayed content to be annotated based on the target lead line information.

[0099] The target lead information is the information of the target lead, so the target lead can be accurately generated based on the target lead information. The target lead can also be displayed in the drawing to be annotated, so as to facilitate the reading of the reader.

[0100] The annotation method of the disclosed embodiment automatically and accurately predicts the target position information and target lead information of the annotation content corresponding to the annotated object in the drawing to be annotated through a pre-trained neural network model, thereby avoiding manual thinking about how to set the position information and lead lines of the annotation content, thereby improving the annotation efficiency.

[0101] In some embodiments of the present disclosure, the target lead information includes a first point, a second point, and a turning point, the first point is a connection point of the target lead on the object to be annotated, the second point is a connection point of the target lead on the target position, the target lead can reach the turning point from the first point along a first direction, and after reaching the turning point, reach the second point along a second direction. The first direction may be different from the second direction.

[0102] In this way, the target lead can be accurately determined based on the target lead information, and the target lead is a broken line, which can appropriately avoid other contents in the drawing, thereby preventing the target lead from blocking other contents of the drawing and improving the readability of the drawing.

[0103] Regarding step S220, in some embodiments of the present disclosure, when the drawing data to be annotated also includes annotated information, step S220 can specifically be: inputting the display drawing information, feature information of the object to be annotated, content to be annotated, annotated information and candidate position information into the trained neural network model, wherein the annotated information includes feature information of the annotated object, known lead information corresponding to the annotated object and known position information of the annotated content corresponding to the annotated object.

[0104] In this way, when there are already marked objects in the drawings to be marked, the neural network model can accurately determine the target position information and target lead information related to the objects to be marked in combination with the already marked objects, thereby saving manpower and improving marking efficiency.

[0105] Regarding step S220, in some embodiments of the present disclosure, when the drawing data to be annotated does not include annotated information, step S220 can specifically be: inputting the display image information, feature information of the object to be annotated, content to be annotated, a preset empty list and candidate position information into the trained neural network model.

[0106] In this way, when there is no annotated object in the drawing to be annotated, the neural network model can also run normally, so as to determine the target position information and target lead information related to the object to be annotated, saving manpower and improving annotation efficiency.

[0107] Based on any of the above embodiments, the present disclosure also provides a model training device.

[0108] Figure 7 It is a schematic block diagram of the structure of a model training device according to an embodiment of the present invention.

[0109] like Figure 7 As shown, the model training device includes: The first determination module 110 is used to determine the annotated drawing data, which includes display drawing information, feature information of the first annotated object in the display drawing, first annotation content and first lead information corresponding to the first annotated object, first position information of the first annotation content, feature information of the second annotated object in the display drawing, second lead information corresponding to the second annotated object, second position information of the second annotation content corresponding to the second annotated object, and candidate position information.

[0110] The first prediction module 120 is used to input the display image information, the feature information of the first annotated object, the first annotated content, the feature information of the second annotated object, the second lead information, the second position information and the candidate position information into the neural network model, and predict the position information of the first annotated content and the lead information corresponding to the first annotated object through the neural network model to obtain the predicted position information and the predicted lead information.

[0111] The first calculation module 130 is used to calculate a target loss value based on the first position information, the first lead information, the predicted position information and the predicted lead information.

[0112] The adjustment module 140 is used to adjust the parameters of the neural network model based on the target loss value to obtain a trained neural network model, wherein the trained neural network model can predict the target position information and target lead information of the annotation content corresponding to the annotated object in the drawing to be annotated.

[0113] The above-mentioned model training device can be in the form of computer software, and each module of the above-mentioned model training device can be implemented by a computer software module.

[0114] In some embodiments of the present disclosure, the first calculation module 130 is used to: determine a first loss value based on a difference between the first position information and the predicted position information; determine a second loss value based on a difference between the first lead information and the predicted lead information; and determine a target loss value based on at least the sum of the first loss value and the second loss value.

[0115] In some embodiments of the present disclosure, the first loss value is obtained by a first cross entropy loss function, and the second loss value is obtained by a second cross entropy loss function.

[0116] In some embodiments of the present disclosure, the model training device also includes: a second determination module, used to determine one or more of the number of lead intersections, angle difference and length in the display diagram based on the predicted lead information, wherein the number of lead intersections is the number of times the predicted lead corresponding to the predicted lead information and the second lead corresponding to the second lead information cross each other, the angle difference is the difference between the angle between the predicted lead and the reference line relative to the target expected angle, the target expected angle is the expected angle corresponding to the angle interval where the angle is located, and the length in the display diagram is the length of the predicted lead in the display diagram; a second calculation module, used to calculate a third loss value based on one or more of the number of lead intersections, angle difference and length in the display diagram; accordingly, the first calculation module 130 is used to: take the sum of the first loss value, the second loss value and the third loss value as the target loss value.

[0117] In some embodiments of the present disclosure, the second calculation module is used to: determine a first score based on the number of lead intersections, the first score being positively correlated with the number of lead intersections; determine a second score based on the angle difference, the second score being positively correlated with the angle difference; determine a third score based on the length in the display diagram, the third score being positively correlated with the length in the display diagram; and perform weighted summation of the first score, the second score, and the third score to obtain a third loss value.

[0118] In some embodiments of the present disclosure, the display image information includes one or more of a display image boundary and a display image centroid point.

[0119] In some embodiments of the present disclosure, the first labeled object feature information and the second labeled object feature information respectively include one or more of a labeled object type, a labeled object centroid point, and a labeled object boundary.

[0120] In some embodiments of the present disclosure, the first lead information and the second lead information respectively include multiple of the lead connection points on the annotated object, the lead connection points at the positions of the annotation content corresponding to the annotated object, the lead turning points and the lead types.

[0121] Based on any of the above implementations, the present disclosure also provides a marking device.

[0122] Figure 8 It is a schematic block diagram of the structure of a labeling device according to an embodiment of the present disclosure.

[0123] like Figure 8 As shown, the marking device comprises: The third determination module 210 is used to determine the drawing data to be annotated, which at least includes the display drawing information, the feature information of the object to be annotated in the display drawing, the content to be annotated corresponding to the object to be annotated, and the candidate position information.

[0124] The second prediction module 220 is used to input the drawing data to be annotated into the trained neural network model, and predict the position information of the content to be annotated and the lead information corresponding to the object to be annotated through the neural network model to obtain the target position information and the target lead information, wherein the neural network model is obtained through the model training method of any of the above-mentioned embodiments.

[0125] The display module 230 is used to display the content to be marked at a target position in the drawing to be marked, wherein the target position is obtained through target position information.

[0126] The generating module 240 is used to generate a target lead line connecting the displayed content to be annotated and the object to be annotated corresponding to the displayed content to be annotated based on the target lead line information.

[0127] In some embodiments of the present disclosure, the target lead information includes a first point, a second point and a turning point, the first point is the connection point of the target lead on the object to be marked, the second point is the connection point of the target lead at the target position, and the target lead can reach the turning point from the first point along the first direction, and reach the second point along the second direction after reaching the turning point.

[0128] In some embodiments of the present disclosure, the second prediction module 220 is used to: in response to the drawing data to be annotated also including annotated information, input the display image information, feature information of the object to be annotated, content to be annotated, annotated information and candidate position information into the trained neural network model, wherein the annotated information includes feature information of the annotated object, known lead information corresponding to the annotated object and known position information of the annotated content corresponding to the annotated object; and in response to the drawing data to be annotated not including annotated information, input the display image information, feature information of the object to be annotated, content to be annotated, a preset empty list and candidate position information into the trained neural network model.

[0129] The above-mentioned marking device may be in the form of computer software, and each module of the above-mentioned marking device may be implemented by a computer software module.

[0130] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0131] The executors of the model training method and the labeling method in the specific implementation manner of the present disclosure may be servers, mobile phones, computers and other electronic devices.

[0132] Therefore, based on any of the above embodiments, the present disclosure also provides an electronic device, which can execute the model training method of any of the embodiments described above in the present disclosure.

[0133] Fig. 9 1 is a schematic block diagram of the structure of an electronic device 1000 according to an embodiment of the present disclosure.

[0134] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0135] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connecting line is used in the figure, but it does not mean that there is only one bus or one type of bus. A computer program is stored in the memory 1300, and when the processor 1200 executes the computer program, the processor 1200 can execute the model training method of any of the above embodiments.

[0136] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0137] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0138] A computer program or instruction may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, a computer program or instruction may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0139] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0144] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0145] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A model training method, characterized in that: include: Determine annotated drawing data, wherein the annotated drawing data includes display drawing information, feature information of a first annotated object in the display drawing, first annotation content and first lead information corresponding to the first annotated object, first position information of the first annotation content, feature information of a second annotated object in the display drawing, second lead information corresponding to the second annotated object, second position information of a second annotation content corresponding to the second annotated object, and candidate position information; Inputting the display image information, the first annotated object feature information, the first annotated content, the second annotated object feature information, the second lead information, the second position information, and the candidate position information into a neural network model, and predicting the position information of the first annotated content and the lead information corresponding to the first annotated object through the neural network model to obtain predicted position information and predicted lead information; Calculating a target loss value based on the first position information, the first lead information, the predicted position information, and the predicted lead information; as well as The parameters of the neural network model are adjusted based on the target loss value to obtain a trained neural network model, wherein the trained neural network model can predict the target position information and target lead information of the annotation content corresponding to the annotated object in the drawing to be annotated.

2. The model training method according to claim 1, characterized in that: Calculating a target loss value based on the first position information, the first lead information, the predicted position information, and the predicted lead information includes: determining a first loss value according to a difference between the first position information and the predicted position information; determining a second loss value according to a difference between the first lead information and the predicted lead information; and The target loss value is determined at least according to the sum of the first loss value and the second loss value.

3. The model training method according to claim 2, characterized in that: The first loss value is obtained by a first cross entropy loss function, and the second loss value is obtained by a second cross entropy loss function.

4. The model training method according to claim 2, characterized in that: Before determining the target loss value at least according to the sum of the first loss value and the second loss value, the method further includes: Determine one or more of the number of lead intersections, the angle difference, and the length in the display diagram according to the predicted lead information, wherein the number of lead intersections is the number of times the predicted lead corresponding to the predicted lead information and the second lead corresponding to the second lead information cross each other, the angle difference is the difference between the angle between the predicted lead and the reference line and the target expected angle, the target expected angle is the expected angle corresponding to the angle interval in which the angle is located, and the length in the display diagram is the length of the predicted lead in the display diagram; and Calculating a third loss value according to one or more of the number of lead intersections, the angle difference, and the length in the display diagram; Determining the target loss value at least according to the sum of the first loss value and the second loss value includes: The sum of the first loss value, the second loss value and the third loss value is taken as the target loss value.

5. The model training method according to claim 4, characterized in that: Calculating a third loss value according to one or more of the number of lead intersections, the angle difference, and the length in the display diagram includes: determining a first score according to the number of intersections of the lead lines, wherein the first score is positively correlated with the number of intersections of the lead lines; determining a second score based on the angular difference, the second score being positively correlated with the angular difference; Determine a third score according to the length of the display image, wherein the third score is positively correlated with the length of the display image; and The first score, the second score and the third score are weightedly summed to obtain the third loss value.

6. A marking method, characterized in that: include: Determine the drawing data to be annotated, wherein the drawing data to be annotated includes at least the display drawing information, the feature information of the object to be annotated in the display drawing, the content to be annotated corresponding to the object to be annotated, and the candidate position information; Inputting the drawing data to be annotated into a trained neural network model, predicting the position information of the content to be annotated and the lead information corresponding to the object to be annotated by the neural network model to obtain target position information and target lead information, wherein the neural network model is obtained by the model training method described in any one of claims 1 to 5; Displaying the content to be annotated at a target position in the drawing to be annotated, wherein the target position is obtained through the target position information; and Based on the target lead information, a target lead is generated to connect the displayed content to be annotated and the object to be annotated corresponding to the displayed content to be annotated.

7. The marking method according to claim 6, characterized in that: Inputting the drawing data to be annotated into the trained neural network model includes: In response to the drawing data to be annotated also including annotated information, inputting the display drawing information, the feature information of the object to be annotated, the content to be annotated, the annotated information and the candidate position information into the trained neural network model, wherein the annotated information includes the feature information of the annotated object, the known lead information corresponding to the annotated object and the known position information of the annotated content corresponding to the annotated object; and In response to the fact that the to-be-annotated drawing data does not include the annotated information, the display image information, the feature information of the to-be-annotated object, the to-be-annotated content, the preset empty list and the candidate position information are input into the trained neural network model.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images