Image recognition method based on deep learning

Abstract

The present invention relates to an image recognition method based on deep learning, comprising: S1, obtaining corner point information and character information of a floor plan to be recognized, and inputting the corner point information into a quadratic programming model to obtain connection information of wall lines and doors and windows; the quadratic programming model pre-establishes a model including a plurality of corresponding corner point connection rules and constraint conditions of the positional relationship between wall lines and doors and windows; S2, inputting the connection information of wall lines and doors and windows into a graph theory optimization model to obtain regional composition information of each room area in the floor plan; the graph theory optimization model is a pre-established Graham algorithm model improved based on graph theory knowledge; S3, matching the character information with the regional composition information to obtain a recognition result. The method solves the problems in the prior art that the floor plan recognition result is not accurate, the recognition process is complex in calculation and has poor scalability.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image recognition method based on deep learning. Background Art

[0002] In the field of smart home design, with the rapid development of Internet technology and artificial intelligence technology, the demand for various online experiences has increased sharply. Whether in terms of automatic layout or 3D reconstruction, the automatic extraction of floor plan elements has important practical significance. The understanding of information in the architectural floor plan can accurately express the geometric and semantic information of the real scene. These geometric and semantic information usually include: room area information, door and window locations, and geometric layout information of objects; therefore, the recognition of geometric and semantic information in floor plan has become a hot topic in current research.

[0003] The understanding of information in the floor plan can be done by manual annotation or by automatic image recognition technology; the manual annotation method usually adopts a purely manual method or simply uses some mapping tools, which has extremely low work efficiency, wastes a lot of manpower and material resources, and is far from meeting the task of mass production. Two methods for automatically identifying floor plans are provided in the prior art. The first method is to use the traditional opencv model to detect the straight line in the floor plan for recognition, and the second method is to use deep learning technology. The disadvantage of the first method is low accuracy, and it is impossible to recognize multiple types of floor plans at the same time. It usually requires manual adjustment of some hyperparameters to recognize, and the versatility is poor; the deep learning technology of the second method is currently the most popular image recognition technology, but the algorithm currently used in floor plan recognition is still lacking. Before floor plan recognition, a lot of preprocessing work needs to be done on the floor plan, and the algorithm has too high requirements on the quality of the floor plan itself; at the same time, due to the single technology, the recognition algorithm has poor real-time performance, high hardware requirements, and weak scalability, and the accuracy cannot meet the user's usage needs. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an image recognition method based on deep learning, which solves the problems in the prior art that the accuracy of floor plan recognition results is low, the recognition process is computationally complex and has poor scalability.

[0006] (II) Technical solution

[0007] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, an embodiment of the present invention provides an image recognition method based on deep learning.

[0009] S1. Obtaining corner point information and character information of the floor plan to be identified, and inputting the corner point information into a quadratic programming model to obtain connection information of wall lines and doors and windows;

[0010] The quadratic programming model is a pre-established model including a plurality of corresponding corner point connection rules and constraint conditions of the positional relationship between the wall line and the doors and windows;

[0011] S2, inputting the connection information of the wall lines and doors and windows into a graph theory optimization model to obtain the area composition information of each room area in the floor plan;

[0012] The graph theory optimization model is a pre-established Graham algorithm model improved based on graph theory knowledge;

[0013] S3. Match the character information with the area composition information to obtain a recognition result.

[0014] Optionally, the step of obtaining corner point information of the floor plan to be identified in S1 includes:

[0015] Input the floor plan to be identified into the trained neural network to obtain corner point information;

[0016] The neural network is a network obtained by training in advance using labeled floor plan data.

[0017] Optionally, the corner point information includes:

[0018] Corner point types of wall lines, corner point types of doors and windows, and corner point location information;

[0019] and / or,

[0020] The recognition result includes: the area composition of the rooms in the floor plan, the room names, the room sizes, the locations of doors and windows, and the scale information.

[0021] Optionally, the step of obtaining character information of the floor plan to be identified in S1 includes:

[0022] The character information in the floor plan to be recognized is recognized based on OCR recognition technology.

[0023] Optionally, the connection information of the wall line and the door and window includes: the connection relationship between two corner points constituting the wall line, and the connection relationship between two corner points constituting the door or window;

[0024] The constraints corresponding to the corner connection rules include:

[0025] If two corner points contain two directions pointing to each other, they can be connected, otherwise they cannot be connected;

[0026] If the direction of the line connecting two corner points is close to any of the four predefined straight lines, they can be connected, otherwise they cannot be connected;

[0027] The constraints corresponding to the positional relationship between the wall line and the doors and windows include:

[0028] The corner connection result of the door or window must be on the wall line. Otherwise, adjust the wall line connection relationship or adjust the door and window connection relationship. The final connection result must ensure that the door and window are above the wall line.

[0029] Optionally, the area composition information includes: a set of points constituting the starting point and the ending point of the clockwise or counterclockwise wall line of the room, starting point information of doors and windows, ending point information of doors and windows, and classification information of doors and windows.

[0030] Optionally, before S1, the step further includes:

[0031] Establish a quadratic programming model including constraints of multiple wall lines, doors, windows, and corner points, and establish a graph theory optimization model.

[0032] Optionally, in S1, inputting the corner point information into a quadratic programming model to obtain connection information of wall lines and doors and windows includes:

[0033] The corner point information output by the neural network is input into the predefined quadratic programming model, and the loss function in the quadratic programming model is used for iterative optimization to obtain the connection information of the output wall lines and doors and windows;

[0034] Among them, the independent variable of the loss function corresponds to the connection relationship between the corner point variable and the corner point in the constraint condition. If there is a connection relationship, it is defined as 1, otherwise it is defined as 0;

[0035] The iterative optimization process is the process of constantly adjusting the connection between wall lines and doors and windows to ensure that the loss function is minimized;

[0036] and / or,

[0037] The S2 includes:

[0038] Based on the connection information of wall lines, doors and windows, the connection relationship between wall lines, doors and windows is abstracted, and the starting and ending points of wall lines, doors and windows are taken as vertices;

[0039] Improve the Graham algorithm, add the connection relationship between points in the vertex, and define new search rules to form the improved Graham algorithm;

[0040] The connection relationship and vertices of abstract wall lines, doors and windows are input into the improved Graham algorithm to obtain the regional composition information of each room area in the searched floor plan;

[0041] The area composition information includes: coordinate point information of the room area, start and end point information of doors and windows, and classification information of doors and windows.

[0042] In a second aspect, the present invention further provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the deep learning-based image recognition method as described in any one of the first aspects above are implemented.

[0043] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the deep learning-based image recognition method as described in any one of the first aspects above are implemented.

[0044] (III) Beneficial effects

[0045] The beneficial effects of the present invention are as follows: the method of the present invention obtains the corner point information of the wall lines and doors and windows in the floor plan with the help of a neural network, performs secondary planning on the corner point information to obtain the connection information of the wall lines and doors and windows, performs graph theory linear planning on the connection information to obtain specific area information, the starting point and end point information of the doors and windows, and finally optimizes and matches the area information of the floor plan with the character information recognized by OCR, thereby obtaining better recognition results, saving a lot of labor costs, fully meeting the real-time requirements, and most importantly, greatly improving the accuracy.

[0046] In this application, a series of technologies such as deep learning technology, quadratic programming model, graph theory linear optimization model and OCR recognition are adopted. First, a deep neural network model is built in combination with the attention mechanism, and the neural network model is obtained by training a large amount of labeled data. Then, the unknown floor plan is directly input into the trained neural network model to predict the corner point type of the floor plan; secondly, all the corner point types identified by the neural network are input into the defined quadratic programming model, and the connection information between the corner points is obtained through the optimization theory of the quadratic programming model; again, the connection information between the corner points is regionally classified through a large amount of graph theory knowledge in the graph theory optimization model, and the regional composition information including regional classification results and door and window information is obtained; finally, the character information in the unknown floor plan is identified through the OCR recognition technology, and then the position and area name are matched in combination with the previously obtained regional composition information.

[0047] The recognition method of this embodiment can recognize floor plans of various corner point types, and the corner point types can be expanded in the neural network model training stage, thereby having strong scalability in application. In other words, it is convenient to expand the corner point types, and can be used to recognize various types of corner point types (such as straight line corner points, oblique line corner points), and can also recognize floor plans of different styles.

[0048] It should be noted that the attention mechanism model is used when training the neural network, which makes the ability to extract corner features stronger and ultimately greatly improves the recognition accuracy.

[0049] The above-mentioned quadratic programming model is composed of a series of constraints, which are mainly the connection rules of corner points and the positional relationship of wall lines, doors and windows. The quadratic programming model has a very fast planning speed and can be completed in milliseconds, which fully meets real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a process of an image recognition method based on deep learning provided by an embodiment of the present invention;

[0051] Figure 2 The corner point types of wall lines, doors and windows abstracted by the present invention;

[0052] Figure 3 This is a flow chart of deep neural network model training based on the attention mechanism of the present invention;

[0053] Figure 4 A schematic flow chart of a floor plan recognition method based on deep learning provided by another embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0056] Embodiment 1

[0057] like Figure 1 As shown, Figure 1 A flowchart of an image recognition method based on deep learning provided by an embodiment of the present invention is shown. The execution subject of the method of this embodiment is any electronic device with computing and processing capabilities. The method specifically includes the following steps:

[0058] S1. Obtain corner point information and character information of a floor plan to be identified, and input the corner point information into a quadratic programming model to obtain connection information of wall lines and doors and windows.

[0059] For example, the connection information may include a line connecting two end points constituting a wall line, or a line connecting two end points constituting a door or window icon.

[0060] The quadratic programming model is pre-established and includes a plurality of corresponding corner point connection rules and constraint conditions of the positional relationship between the wall line and the doors and windows. For example, the constraint conditions of the corresponding corner point connection rules include: if two directions contained in two corner points point to each other, they can be connected, otherwise they cannot be connected; if the direction of the line connecting the two corner points is similar to any one of the four predefined straight lines, they can be connected, otherwise they cannot be connected;

[0061] The constraints corresponding to the positional relationship between the wall line and the doors and windows include: the corner connection result of the door or window must be on the wall line, otherwise the wall line connection relationship or the door and window connection relationship is adjusted, and the final connection result must ensure that the door and window must be above the wall line.

[0062] For example, the corner point information of the floor plan to be recognized can be obtained through a trained neural network, and the character information in the floor plan can be recognized through OCR recognition technology.

[0063] S2. Inputting the connection information of the wall lines and doors and windows into a graph theory optimization model to obtain the area composition information of each room area in the floor plan.

[0064] For example, the area composition information may include: a set of points constituting the starting point and the ending point of the clockwise or counterclockwise wall line of the room, the starting point information of doors and windows, the ending point information of doors and windows, and the classification information of doors and windows.

[0065] The graph theory optimization model is a pre-established Graham algorithm model improved based on graph theory knowledge.

[0066] S3. Match the character information with the area composition information to obtain a recognition result.

[0067] For example, the recognition result may include: the area composition of the rooms in the floor plan, the room names, the room sizes, the locations of doors and windows, scale information, etc.

[0068] For example, the scale in the floor plan may be obtained in advance, and the obtained scale may be corrected (automatically or manually) to obtain accurate and complete scale information.

[0069] Then, the apartment area and character information are matched, for example, the results of OCR recognition are screened, sorted and corrected; then, optimal matching is performed based on the scale information, apartment area information and OCR sorting results.

[0070] In specific applications, the size of the room area can be obtained through the scale and room area pixel information, and then compared with the room area recognized by OCR. If the difference is too large, it means that the calculated room area is not the corresponding room position recognized by OCR, or it is a common area composed of several rooms (such as a living room or foyer). In this case, the matching information needs to be adjusted.

[0071] The method of this embodiment involves a series of technologies such as deep learning, quadratic programming, graph linear programming and OCR recognition. In the actual application process, it is only necessary to pass the two-dimensional floor plan to be identified through the neural network model of the above example to obtain the corner point information of the wall line and doors and windows in the floor plan, perform secondary programming on the corner point information to obtain the connection information of the wall line and doors and windows, perform graph linear programming on the connection information to obtain specific area information, the starting point and end point information of the doors and windows, and finally optimize and match the area information of the floor plan and the character information recognized by OCR; it has strong scalability and can be used to identify two-dimensional floor plans of various corner point types and styles, saving a lot of labor costs, fully meeting the real-time requirements, and most importantly, the accuracy rate has been greatly improved.

[0072] In an optional manner, when performing the above Figure 1 Before step S1 in the method, the method further includes the following step A1 not shown in the figure:

[0073] A1: Train the neural network model.

[0074] For example, a training set may be obtained, and each floor plan data in the training set may be floor plan data that is manually annotated in advance, such as annotating the corner point types of wall lines and doors and windows.

[0075] The data in the above training set is input into the neural network model with selected parameters for training, and the parameters are continuously adjusted until convergence, thereby obtaining the trained neural network model.

[0076] Each floor plan data in the training set may include:

[0077] Pre-mark the wall lines and doors and windows in the floor plan, and use line segments to represent the wall line and door and window marking data;

[0078] Organize the labeled data and calculate the types of corner points as label data for input into the neural network model.

[0079] In a specific application, the floor plan to be identified is input into the trained neural network model to obtain the corner point information of the wall lines, doors and windows in the floor plan. That is, the floor plan is processed by the neural network model to obtain the corner point type and corner point position information of the wall lines, doors and windows in the floor plan.

[0080] like Figure 2As shown in the figure, the corner points of the wall lines and doors and windows in the floor plan are abstracted into Figure 2 Several types of corner points are shown in . Figure 2 Only some corner point types are shown in the figure. This embodiment does not limit the corner point types. The corner point types can be expanded during the training of the neural network model. Therefore, the method of the present application can better achieve scalability and can recognize floor plans of various styles.

[0081] It should be noted that each floor plan input into the neural network model does not require any preprocessing operation, and the original image can be input.

[0082] Furthermore, if Figure 3 As shown, Figure 3 A schematic diagram of the training process is shown. During the training process, information of various manually annotated data can be first obtained, and then input into the neural network model as a training set for training until the neural network model converges, and then parameter information of the converged neural network model is obtained, and the trained neural network model is output.

[0083] It should be noted that during the training process, the present application can be trained based on the attention mechanism, that is, by quickly scanning the global image, obtaining the target area / attention focus that needs to be focused on, and then investing more attention resources in this area to obtain more detailed information of the target that needs to be paid attention to, while suppressing other useless information.

[0084] During the training process, using limited attention resources to quickly filter out high-value information from a large amount of information can greatly improve the efficiency and accuracy of visual information processing.

[0085] Therefore, using the attention mechanism in the process of training the neural network model can improve the processing speed and accuracy of the neural network model.

[0086] In the second optional method, when executing the above Figure 1 Before step S1 in the method, the method further includes the following step A2 which is not shown in the figure:

[0087] A2: Establish a quadratic programming model.

[0088] That is to say, a quadratic programming model is defined, which consists of a series of constraints, which are mainly the connection rules of corner points and the positional relationships of wall lines, doors and windows.

[0089] For example, the definition of the corner point connection rule: Here, it is explained that the corner point contains at most 8 direction types (M-shaped), each corner point contains some of the 8 directions, and each corner point can only be connected in four directions: horizontal, vertical, 45-degree diagonal, and 135-degree diagonal.

[0090] In this embodiment, the corner point connection rules include: if two directions contained in the two corner points point to each other, they can be connected, otherwise they cannot be connected; if the direction of the line connecting the two corner points is similar to any one of the four predefined straight lines, they can be connected, otherwise they cannot be connected; the position relationship between the wall line and the doors and windows is defined as: the corner point connection result of the door or window must be on the wall line, otherwise the wall line connection relationship or the door and window connection relationship is adjusted, and the final result must ensure that the door and window must be above the wall line.

[0091] When the quadratic programming model is applied, the corner point information output by the above neural network is input into the pre-established quadratic programming model, and the connection information of the wall line and the doors and windows is directly output through iterative optimization.

[0092] Among them, the independent variable of the loss function in the quadratic programming model is the connection relationship between the corner point variable and the corner point in the above constraint condition (1 if connected, otherwise 0), and the loss function is a loss value that measures the failure of the corner point to be correctly connected. Iterative optimization process: the process of constantly adjusting the connection method of the wall line and the doors and windows to ensure the minimum loss function. The iterative optimization process in this embodiment is extremely short, averaging 0.003 seconds, thereby meeting the real-time requirements in the recognition process. That is, the quadratic programming model in this embodiment has a very fast planning speed in the process of processing the floor plan, which can be completed in milliseconds, improving the user experience.

[0093] In a third alternative, when executing the above Figure 1 Before step S2 in the method, the method further includes the following step A3 not shown in the figure:

[0094] A3: Establish a graph theory optimization model.

[0095] The graph theory optimization model is a pre-established Graham algorithm model improved based on graph theory knowledge.

[0096] The basic idea of ​​a simple Graham algorithm model is to first find a point on the convex hull, and then start from that point to find points on the convex hull one by one in a counterclockwise direction. In fact, it is to sort the polar angles and then use them for query. Here, in this embodiment, the points scanned by the Graham algorithm model include not only the coordinate information of the points but also the connection information between the points. When searching for another point through one point, it is necessary to consider whether there is a connection relationship between the two points. Points without a connection relationship cannot participate in the search.

[0097] Accordingly, the step of S2 may include:

[0098] First, the abstract wall lines and doors and windows are connected, and the starting and ending points of the wall lines and doors and windows are vertices.

[0099] Secondly, according to the basic principle of Graham algorithm, the connection relationship between points is added to the vertices, and the search rules are redefined to form a new Graham algorithm; that is, the improvement of the existing Graham algorithm includes adding the connection relationship between points to the vertices and redefining the search rules.

[0100] Furthermore, the abstracted wall line and door and window data (such as connection relationships and vertices, etc.) are input into the new Graham algorithm to return the coordinate point information of the room area and the starting and ending point information of the doors and windows, as well as the classification information of the doors and windows.

[0101] It is understandable that the specific implementation steps of the new Graham algorithm may include:

[0102] L1. First find the convex hull point of the wall point set;

[0103] L2. Randomly select a point from the convex hull points (the convex hull points must be on the outer contour);

[0104] L3. In clockwise direction, search for the leftmost straight line and add it, iterate the endpoints (i.e. the starting point and / or end point of the straight line). In this process, if the other end point of a straight line is not connected to other straight lines (i.e. it extends outward from a point on the outer wall to form a separate wall), the straight line is added, the endpoint is returned, and the search continues along other straight lines where the endpoint is located until the search point returns to the origin. At this point, the path traversed is the outermost contour.

[0105] Alternatively, when searching the room area, you can search in a counterclockwise direction and follow the straight line to find the leftmost straight line and add it, and iterate the endpoints. During this process, if the other endpoint of a straight line is not connected to other straight lines (that is, it extends outward from a certain point on the outer wall to form a separate wall), the straight line is added, the endpoint is returned, and the search is continued along the other straight lines where the endpoint is located until the search point returns to the origin.

[0106] It should be noted that the points in the embodiments of the present application include: Figure 2 The corner point information in the corner point type shown in the figure, the vertex refers to the two-dimensional point coordinates in the two-dimensional floor plan, including Figure 2 The two-dimensional point coordinates of the corner points shown in .

[0107] The method of this embodiment can be used to identify various types of two-dimensional floor plans, saving a lot of labor costs, fully meeting real-time requirements, and most importantly, greatly improving the accuracy.

[0108] Embodiment 2

[0109] like Figure 4 As shown, the image recognition method based on deep learning in this embodiment may include the following steps:

[0110] Step 01: Input the two-dimensional floor plan to be identified into the trained neural network, and then obtain the corner point information.

[0111] It should be noted that the trained neural network can be Figure 3 The neural network obtained by training.

[0112] Step 02: Input the corner point information output by the neural network into the quadratic programming model to obtain the connection information model between the corner points.

[0113] The quadratic programming model in this embodiment may be a model including multiple constraint conditions.

[0114] Step 03: Input the connection information into the graph theory optimization model to obtain the area composition information corresponding to the connected area information and the door and window information, that is, the area composition information of the room.

[0115] It should be noted that the graph theory optimization model is a pre-established Graham algorithm improved based on graph theory knowledge.

[0116] Step 04: Input the two-dimensional floor plan to be recognized into the OCR recognition model to obtain the character information in the two-dimensional floor plan;

[0117] Step 05: Match the room's area composition information with the character information in the floor plan and output the final recognition result.

[0118] The method of this embodiment can be used to identify various types of two-dimensional floor plans, saving a lot of labor costs, fully meeting real-time requirements, and most importantly, greatly improving the accuracy.

[0119] It should be noted that, in step 05, before matching the area composition information of the room with the character information in the floor plan, the scale information in the recognized character information may be automatically corrected. The specific correction process may be described as follows:

[0120] Step P1: For the floor plan to be processed, obtain character information of all characters in the floor plan using OCR recognition.

[0121] In an embodiment, the character information may include: text characters, position information of text characters, numeric characters, position information of numeric characters, area information of each room, etc.

[0122] Step P2: Segment a scale area from the floor plan, and obtain a scale value S1 corresponding to the first processing method according to the character information and the scale area.

[0123] For example, step P2 may include the following sub-steps:

[0124] P2-1. According to the distribution characteristics of the scale information, a regional image containing the scale information is segmented in the floor plan as a scale area, and character information belonging to the scale area in the character information is converted according to the scale area to obtain character information of the converted scale area.

[0125] Usually, the scale in the floor plan is drawn around the floor plan. The background of the area is generally monotonous, with few complex interferences. It is composed of line segments and the actual length corresponding to the line segments.

[0126] To this end, in this embodiment, the surrounding area can be segmented from the floor plan as the scale area.

[0127] P2-2. Combining the character information of the converted scale area, identifying the scale mark in the scale area; and obtaining the scale value S1 according to the identified scale mark.

[0128] Step P3: Obtain a scale value S2 corresponding to the second processing method according to the room area information in the floor plan.

[0129] For example, step P3 may be specifically implemented as follows:

[0130] P3-1, according to the number of pixels in each connected region and the actual area information of the connected region, use formula 2 to obtain an array of scale value 2;

[0131] Formula 2 is: Scale value 2 = sqrt (actual area size of the region / total number of pixels in the region);

[0132] Each connected area is a room area predetermined in the floor plan, and the actual area information of the connected area is the area information of each room in the character information;

[0133] P3-2. Use the Grubbs criterion method to remove abnormal values ​​in an array of scale value two, calculate the average value of the filtered data, and obtain the scale value two in the floor plan.

[0134] Step P4: Use the scale value 2 to correct the scale value 1 to obtain a final scale value.

[0135] Specifically, the calibration process may include: determining whether scale value 1 and scale value 2 are both non-zero, and if scale value 1 is zero, taking scale value 2 as the final scale value; if scale value 2 is zero, taking scale value 1 as the final scale value;

[0136] Otherwise, determine whether the error between scale value 1 and scale value 2 is less than a preset threshold. If so, use scale value 1 as the final scale value. Otherwise, re-acquire scale value 1 and repeat the determination process in P4.

[0137] The method of this embodiment uses image processing technology, data precision processing technology, data correction method and OCR recognition technology to process the floor plan. It uses OCR technology and image technology to obtain scale value one, uses the information of the rooms in the floor plan to obtain scale value two, and then corrects scale value one according to scale value two. In this way, a scale value with higher precision can be better obtained, and the character information can be corrected using a high-precision scale value. In this way, the recognition result of the floor plan can be more accurate.

[0138] Embodiment 3

[0139] in addition, Figure 5 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The mobile device may be a mobile terminal, an IPAD, a computer, etc. Figure 1 and Figure 4 The described method may be implemented by an electronic device.

[0140] Figure 5 The mobile device shown may include: at least one processor 61, at least one memory 62, at least one network interface 64 and other user interfaces 63. The various components in the mobile device are coupled together via a bus system 65. It is understood that the bus system 65 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 65 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 65 is not shown in FIG. Figure 5 Various buses are labeled as bus system 65 .

[0141] The user interface 63 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, or a touch pad).

[0142] It is understood that the memory 62 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. In some embodiments, the memory 62 stores the following elements, executable units or data structures, or their subsets, or their extended sets: an operating system 621 and an application 622.

[0143] The operating system 621 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 622 includes various application programs, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 622.

[0144] In the embodiment of the present invention, the processor 61 calls the program or instruction stored in the memory 62, specifically, the program or instruction stored in the application 622, and the processor 61 is used to execute the method steps provided in the first aspect, for example, including the following steps:

[0145] Obtaining corner point information and character information of the floor plan to be identified, and inputting the corner point information into the quadratic programming model to obtain connection information of wall lines and doors and windows;

[0146] The quadratic programming model is pre-established and includes a plurality of corresponding corner point connection rules and constraint conditions of the positional relationship between the wall line and the doors and windows;

[0147] Inputting the connection information of the wall lines and doors and windows into a graph theory optimization model to obtain the area composition information of each room area in the floor plan;

[0148] The graph theory optimization model is a pre-established Graham algorithm improved based on graph theory knowledge;

[0149] The character information and the area composition information are matched to obtain a recognition result.

[0150] The method disclosed in the above embodiment of the present invention can be applied to the processor 61, or implemented by the processor 61. The processor 61 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 61 or the instructions in the form of software. The above processor 61 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 62, and the processor 61 reads the information in the memory 62 and completes the steps of the above method in combination with its hardware.

[0151] In addition, in combination with the deep learning-based image recognition method in the above-mentioned embodiment, an embodiment of the present invention may provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any one of the deep learning-based image recognition methods in the above method embodiments is implemented.

[0152] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP Devices, DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0153] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0154] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0155] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0156] In the embodiments provided in the present application, it should be understood that unless there is a clear sequence between the steps of the method embodiments, the execution order can be adjusted arbitrarily. The disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0159] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present invention.

[0160] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0161] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0162] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An image recognition method based on deep learning, characterized in that: include: S1. Input the floor plan to be identified into the trained neural network to obtain corner point information, input the corner point information into a predefined quadratic programming model, perform iterative optimization with the aid of the loss function in the quadratic programming model, obtain output wall line and door and window connection information, and obtain character information of the floor plan to be identified; The neural network is a network obtained by pre-training with labeled floor plan data, and the quadratic programming model is a pre-established model including a plurality of corresponding corner point connection rules and constraint conditions of the positional relationship between the wall line and the doors and windows; The independent variable of the loss function corresponds to the connection relationship between the corner point variable and the corner point in the constraint condition. If there is a connection relationship, it is defined as 1, otherwise it is defined as 0; The iterative optimization process is a process of continuously adjusting the connection mode of the wall line and the doors and windows to ensure that the loss function is minimized; S2, inputting the connection information of the wall lines and doors and windows into a graph theory optimization model to obtain the area composition information of each room area in the floor plan; The graph theory optimization model is a pre-established Graham algorithm model improved based on graph theory knowledge; S3. Match the character information with the area composition information to obtain a recognition result.

2. The method according to claim 1, characterized in that The corner point information includes: The corner point type of the wall line, the corner point type of the door and window, and the location information of the corner point; and / or, The recognition result includes: the area composition of the rooms in the floor plan, the room names, the room sizes, the locations of doors and windows, and the scale information.

3. The method according to claim 1, characterized in that The step of obtaining character information of the floor plan to be identified in S1 includes: The character information in the floor plan to be recognized is recognized based on OCR recognition technology.

4. The method according to claim 1, characterized in that The connection information of the wall line and the door and window includes: the connection relationship between the two corner points constituting the wall line, and the connection relationship between the two corner points constituting the door or window; The constraints corresponding to the corner connection rules include: If two corner points contain two directions pointing to each other, they can be connected, otherwise they cannot be connected; If the direction of the line connecting two corner points is close to any of the four predefined straight lines, they can be connected, otherwise they cannot be connected; The constraints corresponding to the positional relationship between the wall line and the doors and windows include: The corner connection result of the door or window must be on the wall line. Otherwise, adjust the wall line connection relationship or adjust the door and window connection relationship. The final connection result must ensure that the door and window are above the wall line.

5. The method according to claim 1, characterized in that The area composition information includes: a set of points constituting the starting point and the ending point of the clockwise or counterclockwise wall line of the room, the starting point information of the doors and windows, the ending point information of the doors and windows, and the classification information of the doors and windows.

6. The method according to claim 1, characterized in that Before S1, it also includes: Establish a quadratic programming model including constraints of multiple wall lines, doors, windows, and corner points, and establish a graph theory optimization model.

7. The method according to claim 6, characterized in that The S2 includes: Based on the connection information of wall lines, doors and windows, the connection relationship between wall lines, doors and windows is abstracted, and the starting and ending points of wall lines, doors and windows are taken as vertices; Improve the Graham algorithm, add the connection relationship between points in the vertex, and define new search rules to form the improved Graham algorithm; The connection relationship and vertices of abstract wall lines, doors and windows are input into the improved Graham algorithm to obtain the regional composition information of each room area in the searched floor plan; The area composition information includes: coordinate point information of the room area, start and end point information of doors and windows, and classification information of doors and windows.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the deep learning-based image recognition method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the deep learning-based image recognition method as described in any one of claims 1 to 7 are implemented.

Images