Mind map recognition method, device, equipment, medium and program product

Abstract

The present application discloses a mind map recognition method, apparatus, device, medium, and program product. The method includes: obtaining a target image containing a target mind map; in response to a restoration operation on the target mind map, displaying a predicted root node in the target mind map in the target image; and in response to a confirmation operation on the predicted root node, outputting the target mind map, wherein the target mind map is in an editable state. The present application can automatically identify the root node in a mind map, thereby improving the intelligence and accuracy of mind map restoration.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a mind map recognition method, device, equipment, medium and program product. Background Art

[0002] Mind mapping is a graphical tool that can effectively express divergent thinking. Using mind maps allows for efficient analysis and organization of vast amounts of information, helping users clarify relationships between them. However, mind maps often exist in the form of images, making them difficult to edit. Therefore, restoring mind maps has become a hot topic of research.

[0003] Practice has shown that existing methods for restoring mind maps include: 1. Users manually mark the root node of the mind map in the image, and then restore the mind map based on the manually marked root node. This increases the user's workload and requires manual intervention throughout the mind map restoration process, making it less automated and intelligent, and resulting in a poor user experience. 2. Restoring mind maps using a priori rules, such as specifying the node at absolute coordinates (such as (0, 0)) as the root node to restore the mind map. This method often selects the wrong root node and can only be used in specific scenarios (for example, requiring the user to place the root node at a specified location), resulting in poor generalization. Summary of the Invention

[0004] The embodiments of the present application provide a mind map recognition method, apparatus, device, medium, and program product, which can automatically identify the root node in a mind map and improve the intelligence and accuracy of mind map restoration.

[0005] In one aspect, an embodiment of the present application provides a mind map recognition method, the method comprising:

[0006] Get a target image containing a target mind map;

[0007] In response to a restoration operation on the target mind map, displaying a predicted root node in the target mind map in the target image;

[0008] In response to a confirmation operation on the predicted root node, a target mind map is output, and the target mind map is in an editable state.

[0009] On the other hand, an embodiment of the present application provides a mind map recognition device, the device comprising:

[0010] an acquisition unit, configured to acquire a target image containing a target mind map;

[0011] a processing unit, configured to display the predicted root node in the target mind map in the target image in response to a restoration operation on the target mind map;

[0012] The processing unit is further configured to output a target mind map in response to a confirmation operation on the predicted root node, where the target mind map is in an editable state.

[0013] In one implementation, when the processing unit is configured to display the predicted root node in the target mind map in the target image, it is specifically configured to:

[0014] Annotate the target image with the predicted root node in the target mind map;

[0015] The annotation display includes: displaying the predicted root node in the form of a visual display; or displaying the predicted root node in the form of an annotation mark;

[0016] The annotation mark is displayed in the area where the predicted root node is located; or, the annotation mark is displayed in the target image in the form of an annotation.

[0017] In one implementation, when the processing unit is used to obtain a target image including a target mind map, it is specifically used to:

[0018] Display the function selection interface, which includes the option of converting images to mind maps;

[0019] In response to triggering the image-to-mind-map option, displaying an image acquisition interface;

[0020] In the image acquisition interface, a target image containing a target mind map is acquired.

[0021] In one implementation, the image acquisition interface includes an image upload option, and the processing unit is configured to acquire a target image including a target mind map in the image acquisition interface by:

[0022] In response to triggering the image upload option, displaying at least one candidate image;

[0023] Any candidate image is selected from at least one candidate image, the selected candidate image is used as a target image, and a mind map included in the candidate image is used as a target mind map included in the target image.

[0024] In one implementation, the image acquisition interface includes an image scanning option, and the processing unit is configured to acquire a target image including a target mind map in the image acquisition interface by:

[0025] In response to triggering the image scanning option, a scanning operation is performed on the target image containing the target mind map to obtain the target image.

[0026] In one implementation, the processing unit is further configured to:

[0027] In response to an operation of reselecting a root node in the target image, canceling the labeling display of the predicted root node in the target image, and labeling and displaying the reselected new root node in the target image;

[0028] In response to a confirmation operation on the new root node, outputting a target mind map;

[0029] The annotation display of the predicted root node is revoked to indicate that the predicted root node has not been confirmed.

[0030] In one implementation, the processing unit is further configured to:

[0031] In response to an editing operation on a target mind map, updating the target mind map;

[0032] In response to a sharing operation on the updated target mind map, a sharing process is performed on a sharing image including the updated target mind map.

[0033] In one implementation, the processing unit is configured to, in response to a restoration operation on a target mind map, display a predicted root node in the target mind map in a target image, specifically:

[0034] In response to the restoration operation on the target mind map, calling the target semantic segmentation model to perform root node prediction processing on the target image to obtain an initial predicted root node in the target mind map;

[0035] According to the initial prediction root node and each node in the target mind map, determining the node distance between the initial prediction root node and each node;

[0036] The node with the shortest distance to the initial predicted root node is used as the predicted root node.

[0037] In one implementation, the processing unit is configured to call the target semantic segmentation model to perform root node prediction processing on the target image, and obtain an initial predicted root node in the target mind map, specifically for:

[0038] Call the target semantic segmentation model to perform semantic segmentation processing on the target image to obtain a target semantic label image corresponding to the target image, where the target semantic label image includes the foreground area;

[0039] The target connected domain in the foreground area is regarded as the salient area in the target image;

[0040] The center point of the salient area is used as the initial predicted root node in the target mind map.

[0041] In one implementation, the processing unit is further configured to:

[0042] Acquire a training image, wherein the training image includes a training mind map;

[0043] Performing a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image; and

[0044] Calling the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantic label image;

[0045] Obtaining a loss function of the initial semantic segmentation model, and calculating a loss value of the loss function based on the first semantic label image and the second semantic label image;

[0046] If the loss value meets the preset conditions, the initial semantic segmentation model reaches the convergence condition and is used as the trained target semantic segmentation model.

[0047] If the loss value does not meet the preset conditions, the initial semantic segmentation model is iteratively trained.

[0048] In one implementation, the processing unit is configured to perform a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image, specifically for:

[0049] Performing region extraction processing on the training image to obtain an attention image, which includes a salient region;

[0050] Extract features from the attention image to obtain a feature map;

[0051] The category labels in the feature map are redefined according to the attention image to obtain a first semantic label image.

[0052] In one implementation, the processing unit is configured to perform region extraction processing on the training image to obtain the attention image, specifically for:

[0053] Obtaining a root node annotation file corresponding to the training image, wherein the root node annotation file includes coordinate information of the root node in the preset training mind map;

[0054] A region extraction process is performed on the training image based on the coordinate information to generate an attention image, wherein the attention image includes a salient region centered at a coordinate position indicated by the coordinate information.

[0055] In one implementation, the salient region refers to a circular region centered at a coordinate position indicated by the coordinate information; the processing unit is configured to perform region extraction processing on the training image based on the coordinate information to obtain the attention image, specifically for:

[0056] Obtain the display size information of the training image and calculate the radiation radius based on the display size information;

[0057] Determine the salient area in the training image based on the coordinate information and radiation radius;

[0058] The thermal value of each feature point in the salient region is calculated to generate an attention image; the thermal value of the feature point in the salient region in the attention image is greater than the thermal value threshold.

[0059] In one implementation, the processing unit is configured to redefine the category labels in the feature map according to the attention image to obtain the first semantic label image, specifically to:

[0060] The thermal value of each feature point in the salient area of the attention image is used to replace the label value of the category label of the corresponding feature point in the feature map to obtain a first semantic label image.

[0061] In one implementation, the processing unit is configured to call the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain the second semantic label image, specifically for:

[0062] Call the downsampling module in the initial semantic segmentation model to downsample the training image to obtain the first resolution image;

[0063] Call the N-order convolutional layer in the initial semantic segmentation model to extract features from the first resolution image to obtain N feature maps with different resolutions; N is an integer greater than 1;

[0064] Call the feature fusion module in the initial semantic segmentation model to perform feature fusion processing on N feature maps to obtain the target feature map;

[0065] The classification module in the initial semantic segmentation model is called to classify the target feature map to obtain a second semantic label image.

[0066] In one implementation, the processing unit is configured to call a feature fusion module in the initial semantic segmentation model to perform feature fusion processing on N feature maps to obtain a target feature map, specifically for:

[0067] Upsample the feature maps whose resolution is less than the resolution threshold in the N feature maps to obtain N feature maps with the same resolution;

[0068] N feature maps with the same resolution are fused to obtain the target feature map.

[0069] In another aspect, the present application provides a computer device, comprising:

[0070] a processor for loading and executing computer programs;

[0071] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned mind map recognition method.

[0072] On the other hand, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing a mind map recognition method.

[0073] In another aspect, the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described mind map recognition method.

[0074] In an embodiment of the present application, the computer device, in response to the restoration operation on the target mind map, can predict the root node of the target mind map in the target image and display the predicted root node of the target mind map in the target image; this can achieve automatic prediction of the root node in the target mind map, and compared with manual annotation of the root node, it avoids manual intervention in the confirmation process of the root node in the mind map, thereby reducing the workload of the user object. Furthermore, the computer device, in response to the user object's confirmation operation on the predicted root node, can continue to restore the complete target mind map based on the predicted root node; this restoration of the target mind map based on the accurate predicted root node obtained by prediction improves the accuracy of the restored target mind map. In the above scheme, in the restoration process of the target mind map, the complete target mind map can be quickly and accurately restored without manual intervention, thereby improving the restoration efficiency of the target mind map. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0076] Figure 1a A schematic diagram of a mind map provided by an exemplary embodiment of the present application is shown;

[0077] Figure 1bA schematic diagram of a mind map provided by an exemplary embodiment of the present application is shown;

[0078] Figure 2 A schematic diagram of the architecture of a mind map recognition system provided by an exemplary embodiment of the present application is shown;

[0079] Figure 3 A flowchart of a mind map recognition method provided by an exemplary embodiment of the present application is shown;

[0080] Figure 4 A schematic diagram of triggering and displaying an image acquisition interface provided by an exemplary embodiment of the present application is shown;

[0081] Figure 5a A schematic diagram showing an exemplary embodiment of the present application providing a method for obtaining a target image from a storage space through an image upload option is shown;

[0082] Figure 5b A schematic diagram of acquiring a target image by shooting is shown in an exemplary embodiment of the present application;

[0083] Figure 6a A schematic diagram of a restoration operation provided by an exemplary embodiment of the present application is shown;

[0084] Figure 6b A schematic diagram of a restoration operation provided by an exemplary embodiment of the present application is shown;

[0085] Figure 6c A schematic diagram of a restoration operation provided by an exemplary embodiment of the present application is shown;

[0086] Figure 7a A schematic diagram of an exemplary embodiment of the present application providing a method for marking and displaying a predicted root node in a target image is shown;

[0087] Figure 7b A schematic diagram of an exemplary embodiment of the present application providing a method for marking and displaying a predicted root node in a target image is shown;

[0088] Figure 7c A schematic diagram of an exemplary embodiment of the present application providing a method for marking and displaying a predicted root node in a target image is shown;

[0089] Figure 8 A schematic diagram showing an exemplary embodiment of the present application providing a method of outputting a target mind map by triggering a confirmation operation in an image preview interface;

[0090] Figure 9 A flowchart of a mind map recognition method provided by an exemplary embodiment of the present application is shown;

[0091] Figure 10 A schematic diagram of determining a predicted root node in a target mind map provided by an exemplary embodiment of the present application is shown;

[0092] Figure 11 A schematic diagram showing an initial prediction of the Euclidean distance between a root node and each node provided by an exemplary embodiment of the present application is shown;

[0093] Figure 12 A schematic diagram of reselecting a new root node provided by an exemplary embodiment of the present application is shown;

[0094] Figure 13a A schematic diagram of scaling a target image provided by an exemplary embodiment of the present application is shown;

[0095] Figure 13b A schematic diagram of scaling a target image provided by an exemplary embodiment of the present application is shown;

[0096] Figure 13c A schematic diagram of scaling a target image provided by an exemplary embodiment of the present application is shown;

[0097] Figure 13d A schematic diagram of a sliding target image provided by an exemplary embodiment of the present application is shown;

[0098] Figure 14 A schematic diagram of an editing target mind map provided by an exemplary embodiment of the present application is shown;

[0099] Figure 15 A schematic diagram of sharing an edited target mind map provided by an exemplary embodiment of the present application is shown;

[0100] Figure 16 A flowchart of a mind map recognition method provided by an exemplary embodiment of the present application is shown;

[0101] Figure 17 A schematic diagram of determining a first semantic label image corresponding to a training image provided by an exemplary embodiment of the present application is shown;

[0102] Figure 18a A schematic diagram of semantic segmentation provided by an exemplary embodiment of the present application is shown;

[0103] Figure 18b A schematic diagram of semantic segmentation provided by an exemplary embodiment of the present application is shown;

[0104] Figure 19A schematic diagram of training an initial semantic segmentation model provided by an exemplary embodiment of the present application is shown;

[0105] Figure 20 A schematic diagram of a feature fusion process provided by an exemplary embodiment of the present application is shown;

[0106] Figure 21 A schematic structural diagram of a mind map recognition device provided by an exemplary embodiment of the present application is shown;

[0107] Figure 22 A structural diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0108] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0109] The present application embodiment proposes a mind map recognition solution. The following is a brief description of some technical terms and concepts involved in the mind map recognition solution, including:

[0110] 1. Artificial Intelligence (AI)

[0111] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0112] This application primarily relates to machine learning (ML) in artificial intelligence. Machine learning is a multidisciplinary interdisciplinary field encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and model-based learning. Machine learning can be viewed as a task whose goal is to enable machines (in a broad sense, computers) to acquire human-like intelligence through learning. For example, humans can play Go, and a computer program (AlphaGo or AlphaGo Zero) is designed to master Go knowledge and play Go. A variety of methods can be used to implement machine learning tasks, such as neural networks, linear regression, decision trees, support vector machines, Bayesian classifiers, reinforcement learning, probabilistic graphical models, and clustering.

[0113] 2. Mind Map.

[0114] Mind mapping can be called mental mapping, which is a graphic tool that can effectively express divergent thinking. By using mind mapping tools to express complex and rich information in a divergent way, a mind map for representing rich information content can be obtained; user objects can analyze and summarize information through the mind map, which is conducive to quickly sorting out the relationship between each piece of information, making complex information easy to remember and understand. Among them, mind mapping tools may include: exclusive mind mapping applications for supporting the construction or editing of mind maps; or, mind mapping services provided by applications for building or editing mind maps (such as mind mapping services provided by document applications); etc. The above-mentioned applications may refer to computer programs for completing one or more specific tasks. ① According to the classification of the operation mode of the application, the application may include but is not limited to: clients installed in the terminal, small programs that can be used without downloading and installing, web applications opened through the browser, etc. ② According to the functional type of the application, the application may include but is not limited to: IM (Instant Messaging) application, content interaction application or document application, etc.; among them, instant messaging application refers to an application for instant communication of messages and social interaction based on the Internet, and instant messaging application may include but is not limited to: social applications with communication functions, map applications with social interaction functions, game applications, etc. Content interaction application refers to an application that can realize content interaction, such as online banking, sharing platforms, personal space, news and other applications. Document application refers to an application with document editing capabilities, such as online documents, collaborative documents and other applications. The embodiment of this application does not limit the application that supports the construction or editing of mind maps, specifically which type of application is mentioned above, and is specifically explained here.

[0115] Furthermore, the mind map is mainly composed of: root nodes, child nodes and mind map lines. Specifically, the root node is the starting node of the mind map (or called the central node), which is often used to express the core theme (or called the central theme) as the thinking center of the mind map; then, starting from the thinking center, thousands of child nodes can be radiated outward, and each child node represents a connection with the central theme, and each child node can become the next central theme, and then radiate thousands of child nodes outward, presenting a radial three-dimensional structure. It should be understood that the embodiment of the present application does not limit the names of the various nodes and lines in the mind map. For example, the root node can also be called the core node, the child node can also be called the joint point (or leaf node), etc., which is specially explained here. A schematic diagram of an exemplary mind map can be found in Figure 1a ,like Figure 1aAs shown, the mind map starts from root node 101 and radiates outward to the right into four child nodes: child node 1, child node 2, child node 3, and child node 4. Each child node can become the next root node, which then radiates outward to other child nodes. For example, child node 1 acts as a root node, radiating outward to the right to form child nodes 1.1 and 1.2. Similarly, child node 2 acts as a root node, radiating outward to the right to form child nodes 2.1, 2.2, and 2.3, and so on. This process can be used to express complex information in layers, creating a mind map.

[0116] It should be noted that Figure 1a It is just a schematic diagram of an exemplary mind map; Figure 1a In the mind map shown, each node in the mind map is represented by a text block (or image block). However, it is understandable that each node in the mind map can also be represented in other forms, such as the form of line endpoints. An exemplary diagram of a mind map in which nodes are line endpoints can be found in Figure 1b ,like Figure 1b As shown, the root node is a line endpoint, and the core theme represented by the root node can be displayed on the mind map line between the two line endpoints. The embodiment of the present application does not limit the specific style and structure of the mind map.

[0117] In actual applications, the constructed mind map often exists in the form of an image (or picture); when sharing the constructed mind map, an image is generated based on the mind map, and then the image is shared to realize the sharing of the mind map contained in the image. However, the image itself has the characteristic of not supporting the editing of the content contained in the image, which makes it impossible for the user object to perform editing operations (such as adding and deleting nodes, etc.) on the mind map contained in the image. Based on this, when the user object has the need to edit the mind map contained in the image, the mind map contained in the image can be restored based on the restoration technology to restore the mind map that allows editing operations to be performed; in other words, restoring the mind map in the target image means: converting a mind map in an uneditable state into a mind map in an editable state; facilitating the user object to perform editing operations on the mind map in an editable state to realize the update of the mind map, thereby meeting the user object's editing needs for the mind map. As described above, a mind map is a divergent thinking structure that radiates from a root node to the surrounding areas. Therefore, the main idea of the restoration technology used to restore the mind map contained in the image is as follows: first identify the root node of the mind map, then use the root node as the starting node (or simply the starting point), and search for other child nodes related to it in a depth-first traversal manner, advancing layer by layer, and finally restoring the complete structure of the mind map to present an editable mind map to the user.

[0118] Based on the above description, it can be seen that in the process of restoring the mind map, quickly and accurately determining the root node of the mind map is of great significance to restoring the complete mind map; therefore, the embodiment of the present application mainly focuses on the implementation process of identifying or predicting the root node in the mind map. Specifically, considering that the neural network under artificial intelligence has good generalization ability, the embodiment of the present application supports the use of a neural network model (or simply referred to as a neural network) to quickly and accurately identify the root node in the mind map, thereby improving the efficiency and accuracy of restoring the entire mind map. Among them, the neural network model may include a semantic segmentation model (or an image semantic segmentation model), which can be used to classify each pixel in the image to distinguish between significant areas and other areas in the image; the significant areas in the image here may refer to: the areas that human vision pays attention to in the image (or called areas of interest), and other areas may refer to: areas in the image other than significant areas. Of course, the embodiment of the present application does not limit the specific type of network model of the neural network model. For example, other network structures with the same function can be used instead of the semantic segmentation model.

[0119] When humans are presented with an image, their brains automatically activate a visual attention mechanism, automatically processing salient areas within the image. This visual attention mechanism, referred to here as the attention mechanism, is the mechanism by which human vision selectively focuses on a subset of all information (e.g., all information contained in an image), while ignoring other visible information. However, for machine-trained neural networks, what they learn is the feature information (or simply, features) of the entire image. These features are treated identically by the neural network, and it does not overly focus on any particular area. Based on this, the mind map recognition solution proposed in this invention aims to mimic human visual characteristics by rigidly selecting the areas of the input image that require attention, or by assigning different weights to different parts of the input image. This allows the neural network to focus on the areas of interest (e.g., the root node in a mind map) while paying less attention to other information, thereby enabling the neural network to extract salient areas within an image (e.g., the area where the root node is located in a mind map).

[0120] In a specific implementation, the general principle of the mind map recognition solution provided by the embodiment of the present application may include: first, the initial semantic segmentation model (such as the semantic segmentation model to be trained) is trained to obtain a target semantic segmentation model (such as a trained semantic segmentation model). The target semantic segmentation model has good generalization ability and supports the restoration of mind maps of different styles and structures, thereby extending the use scenario of the mind map recognition solution provided by the embodiment of the present application. Then, in response to the user object's restoration requirement for the target mind map in the target image, the target semantic segmentation model is used to perform semantic segmentation processing on the target image to identify the predicted root node in the target mind map; in this way, the root node in the target mind map can be predicted quickly and accurately without manual intervention, thereby improving the restoration efficiency of the target mind map. Secondly, the identified predicted root node is displayed in the target image to facilitate the user object to confirm the predicted root node predicted by the target semantic segmentation model. Finally, in response to the user object's confirmation operation on the predicted root node, the entire target mind map can be restored based on the predicted root node, and the entire target mind map can be output so that the user object can perform editing operations on the restored target mind map.

[0121] In order to better understand the mind map recognition solution provided by the embodiment of this application, the following Figure 2 The mind map recognition system shown in FIG. 1 briefly introduces the mind map recognition scenario involved in the embodiment of the present application; Figure 2As shown, the mind map recognition system includes a computer device 201 and a computer device 202. The embodiment of the present application does not limit the number and naming of the computer device 201 and the computer device 202. Among them, ① the computer device 201 may refer to a terminal deployed with a target application (such as a mind map tool) capable of restoring a mind map. The terminal may include but is not limited to: smart phones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, intelligent voice interaction devices, mobile Internet devices (Mobile Internet Devices, abbreviated as MID), smart home appliances, vehicle-mounted terminals, aircraft, head-mounted devices, and other smart devices that can touch the screen. ② The computer device 202 may refer to the server corresponding to the computer device 201, which is used to provide computing and application support for the computer device 201; the server may include but is not limited to: data processing servers, Web servers, application servers, and other devices with complex computing capabilities. The server can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers. The computer device 201 and the computer device 202 can be directly or indirectly connected to each other via a wired or wireless manner. The embodiment of the present application does not limit the connection method between the computer device 201 and the computer device 202.

[0122] The mind map recognition scheme provided in the embodiment of the present application can be executed by the computer device 201 or the computer device 202 mentioned above, or it can be executed jointly by the computer device 201 and the computer device 202; for the sake of convenience, the following will be introduced by taking the computer device 101 and the computer device 102 jointly executing the mind map recognition scheme provided in the embodiment of the present application as an example. In a specific implementation, the computer device 202 can obtain a training data set for training the initial semantic segmentation model to obtain a trained target semantic segmentation model. The trained target semantic segmentation model can be deployed in the computer device 201, so that when the computer device 201 responds to the restoration operation for the target image containing the target mind map, the target semantic segmentation model is called to perform prediction processing on the target image, so as to quickly and accurately predict the predicted root node in the target mind map. Furthermore, in response to the user object's confirmation operation on the predicted root node predicted by the target semantic segmentation model, the target mind map can continue to be traversed according to the predicted root node to restore the complete target mind map, and the target mind map allows editing operations to be performed. Through the above scheme, it is possible to automatically predict the root node in the target mind map, and then automatically and accurately restore the complete target mind map without manual intervention, reducing the workload of the user object, improving the efficiency of restoring the target mind map, and improving the user object's usage experience.

[0123] It should be noted that, in addition to being deployed on the computer device 201, the target semantic segmentation model trained by the computer device 202 can also be deployed on the computer device 202. Under this implementation, the computer device 202 can receive the target image containing the target mind map sent by the computer device 201, and then call the trained target semantic segmentation model to restore the target mind map; and return the restored complete target mind map to the computer device 201, so that the computer device 201 can output the target mind map, and the user object can edit the restored target mind map.

[0124] Based on the mind map recognition scheme described above, it can be seen that the mind map recognition scheme provided in the embodiment of this application mainly involves two aspects. On the one hand, the model is trained to obtain the target semantic segmentation model, and on the other hand, the trained target semantic segmentation model is used for model application. The following is an introduction to the more detailed mind map recognition method proposed in the embodiment of this application in conjunction with the accompanying drawings.

[0125] Figure 3 A flowchart of a mind map recognition method provided by an exemplary embodiment of the present application is shown; Figure 3 The mind map recognition method shown mainly involves the model application part and can be executed by a computer device (such as the computer device 201). The mind map recognition method may include but is not limited to steps S301-S303:

[0126] S301: Acquire a target image including a target mind map.

[0127] When a user object has a need to restore a target mind map contained in a target image, the user object can use a computer device (such as computer device 201) to obtain a target image containing the target mind map to be restored. Taking a target application (such as a document application) for restoring a mind map deployed in the computer device as an example, in a specific implementation, the user object can open and use the target application to display a function selection interface, which includes an image-to-mind map option; when the user object triggers the image-to-mind map option, it indicates that the user object wants to restore the mind map, and the computer device displays an image acquisition interface in response to the user object triggering the image-to-mind map option; in this way, the user object can input a target image containing the target mind map to be restored in the image acquisition interface, that is, the computer device can acquire the target image containing the target mind map in the image acquisition interface.

[0128] For ease of understanding, the following Figure 4 A schematic diagram of triggering the display of an image acquisition interface is given; Figure 4As shown, in response to the operation of restoring the mind map, the computer device may display a function selection interface 401 provided by the target application, and the function selection interface 401 includes one or more options, and one or more options include an image-to-mind map option 402; when any option is selected (such as clicked), the function corresponding to the selected option may be triggered to execute. In response to the triggering of the image-to-mind map option 402, an image acquisition interface 403 may be displayed, and a target image containing the target mind map may be acquired in the image acquisition interface 403. It should be noted that, depending on the target application with the function of restoring the mind map, the function selection interface and / or image acquisition interface provided by the target application may not contain the same content and style as the one in the image acquisition interface. Figure 4 As shown, Figure 4 This is just an exemplary implementation process of triggering the display of the image acquisition interface from the function selection interface.

[0129] The embodiment of the present application supports the use of multiple implementation methods to obtain the target image containing the target mind map from the image acquisition interface; in other words, more than one path can be used to obtain the target image in the image acquisition interface. For example: the image acquisition interface includes an image upload option (or key, button, control, etc.), which supports uploading the target image in the storage space to the image acquisition interface through the image upload option to obtain the target image; for another example: the image acquisition interface includes an image scanning option, which supports obtaining the target image by scanning through the image scanning option. The following is a more detailed description of the two implementation methods for obtaining the target image given above; among them:

[0130] In one implementation, the target image is acquired through the image upload option in the image acquisition interface. In a specific implementation, the image acquisition interface includes an image upload option. When the user object performs a trigger operation on the image upload option, it indicates that the user object wants to restore the target image in the storage space. The computer device can display at least one candidate image included in the storage space in response to the triggering of the image upload option. Then, the user object can select any candidate image from the at least one candidate image. At this time, the computer device can use any candidate image selected by the user object as the target image, and the mind map contained in any candidate image as the target mind map contained in the target image. Among them, the storage space mentioned above may include: the local storage space of the computer device, or the cloud storage space of the user object, etc.; the embodiment of the present application does not limit the specific type of storage space. The above-mentioned method of acquiring the target image from the storage space is conducive to the user object quickly determining the target image to be restored from the storage space, and improving the user object's restoration experience of the target mind map.

[0131] An exemplary diagram of obtaining a target image from a storage space through an image upload option can be found in Figure 5a ,like Figure 5a As shown, image acquisition interface 501 includes an image upload option 5011. In response to triggering image upload option 502, at least one candidate image, such as candidate image 502, candidate image 503, candidate image 504, etc., may be displayed. In response to selecting any candidate image (such as candidate image 502) from the at least one candidate image, it is determined that the selected candidate image 502 is used as the target image, and the selected target image is displayed in image acquisition interface 501, so that the user object can reconfirm whether to use the displayed candidate image 502 as the target image in image acquisition interface 501. If the user object is not satisfied with the candidate image 502 displayed in the image acquisition interface, the user object can perform the operation of selecting the target image from the candidate images again, thereby satisfying the user object's need to select the target image multiple times.

[0132] In other implementations, the target image is acquired through the image scanning option in the image acquisition interface. In a specific implementation, the image acquisition interface includes an image scanning option. When the user object performs a trigger operation on the image scanning option, it indicates that the user object wants to acquire the target image by scanning. Then, the computer device can respond to the triggering of the image scanning option and perform a scanning operation on the target image containing the target mind map to acquire the target image. After the computer device scans the image and determines that the target image containing the target mind map is acquired, the target image can be displayed in the image acquisition interface to prompt the user object to reconfirm the scanned target image. If the user object is not satisfied with the scanned target image, such as if the scanned target image is unclear or incomplete, the user object can perform the scanning operation again until the acquired target image meets the user object's requirements.

[0133] It should be noted that the target image to be scanned described above can be displayed on a computer device or on a device other than a computer device; the scanning operation performed on the target image will vary depending on the device displaying the target image. Optionally, the target image to be scanned is displayed on the terminal screen of a computer device; in this implementation, the scanning operation on the target image may include: the computer device identifying the target image from the terminal screen; through this method, the image contained in the device can be scanned, enriching the method of acquiring the target image. Optionally, the target image to be scanned is displayed on the terminal screen of a device other than a computer device; in this implementation, the scanning operation on the target image may include: the computer device turning on the camera in response to triggering the image scanning option, and scanning the image on the terminal screen of the other device through the camera. In the case where the target image to be scanned does not belong to the computer device, the above method can be used to jointly acquire the target image through two devices, enriching the method of acquiring the target image and meeting the multi-scenario needs of the user object to acquire the target image.

[0134] Furthermore, when using a computer device to scan a target image containing a target mind map, the conditions for determining whether the computer device has completed the scan may include: the duration of the target image scan is equal to a duration threshold; if the duration threshold is 3 seconds and the detected scan duration is 3 seconds, then the scan is determined to be complete and the scanned image is displayed as the target image in the image acquisition interface. Alternatively, determining whether the scanned image meets image scanning requirements, such as whether the image clarity meets clarity requirements, or whether the content contained in the image is complete. Alternatively, detecting whether a capture button is triggered; if so, determining whether the captured image is displayed as the target image in the image acquisition interface.

[0135] An exemplary schematic diagram of obtaining a target image by shooting can be found in Figure 5b As shown in 5b, the image acquisition interface 501 includes an image upload option 5012; in response to triggering the image upload option 5012, the camera of the computer device (such as a smart phone) can be turned on, and the camera can be aimed at the display screen (or terminal screen) of other devices (such as a smart computer) to achieve the capture of the image displayed on the display screen; when the capture button 505 (or the physical button of the computer device) is triggered, the captured image is determined to be the target image, and the target image is displayed in the image acquisition interface.

[0136] It should be noted that the above are only several exemplary implementation methods of acquiring the target image given in the embodiments of the present application, and the embodiments of the present application do not limit the specific method used to acquire the target image.

[0137] S302: In response to a restoration operation on a target mind map, displaying a predicted root node in the target mind map in a target image.

[0138] Based on the implementation process shown in step S301, after the target image containing the target mind map is acquired in the image acquisition interface, if the user object determines to restore the target mind map contained in the target image displayed in the image acquisition interface, the user object can perform a restore operation in the image acquisition interface. Among them, the restore operation performed by the user object in the image acquisition interface may include but is not limited to: triggering an operation on the completion option; or performing a gesture operation of a preset gesture (such as drawing an "S" shape, an "L" shape, or double-clicking); or inputting a voice input operation for indicating the restore process; etc. The following are several exemplary specific implementation processes of the restore operation, among which:

[0139] Optionally, the restore operation includes a triggering action for the completion option. Figure 6a As shown, the image acquisition interface includes a completion option 601; in response to the triggering operation of the completion option 601, it indicates that the user object confirms to restore the target mind map currently displayed in the image acquisition interface, then it is determined that there is a restoration operation for the target mind map, and the prediction processing of the root node in the target mind map is started.

[0140] Optionally, the restoration operation includes performing a preset gesture (such as drawing an "S" shape). Figure 6b As shown, when the computer device detects that the movement trajectory formed by the movement operation on the terminal screen is "S" shaped, it determines that there is a restoration operation for the target mind map, and then starts to execute the prediction processing of the root node in the target mind map.

[0141] Optionally, the restoration operation includes a voice input operation for inputting a signal for restoration processing. Figure 6c As shown, during the entire process of displaying the image acquisition interface on the terminal screen of the computer device, the recording function of the computer device (such as the microphone is on) is in an on state; then, during the process of displaying the image acquisition interface on the terminal screen, the computer device can collect sound signals in the surrounding environment in real time and identify the collected sound signals. When it is recognized that the content indicated by the sound signal is to restore the target mind map, it is determined that there is a restoration operation for the target mind map, and then the prediction processing of the root node in the target mind map is started.

[0142] Furthermore, in response to the restoration operation on the target mind map, the computer device can predict the root node in the target mind map, and after predicting the predicted root node in the target mind map, switch from the image acquisition interface to the image preview interface; and display the target image in the image preview interface, and display the predicted root node of the located target mind map in the target image. By displaying the predicted root node in the target image, the user object can be intuitively prompted to the position of the predicted root node in the target image, so that the user object can see the position of the predicted root node at a glance in the target image. Of course, the image preview interface and the image acquisition interface can also be the same interface, and the embodiment of the present application does not limit this.

[0143] In a specific implementation, the predicted root node in the target mind map can be annotated and displayed in the target image. The so-called annotated display of the predicted root node may include but is not limited to: displaying the predicted root node in the target image in the form of a visual display; or displaying the predicted root node in the target image in the form of an annotated mark. The following is a detailed description of the exemplary implementation of the above-mentioned annotated display of the predicted root node in conjunction with the accompanying drawings, wherein:

[0144] ① Display the predicted root node in the target mind map in the form of visual display in the target image. The visual display can be called visual highlighting or visual highlighting, and its purpose is to highlight or highlight the predicted root node to intuitively prompt the user of the display position of the predicted root node predicted by the object. The visual display of the predicted root node in the target mind map may include: the color of the displayed predicted root node is brighter or more vivid than the colors of other nodes in the target mind map. Figure 7a As shown, assuming that the color of each node in the target mind map is white, after the predicted root node in the target mind map is predicted, the display color of the predicted root node can be changed to gray or black to visually distinguish the predicted root node from other nodes, so that the user object can intuitively see the position of the predicted root node. Alternatively, the display size of the displayed predicted root node is larger or smaller than the display size of other nodes in the target mind map. Alternatively, the predicted root node is dynamically displayed, while the other nodes in the target mind map are statically displayed to distinguish the various nodes in the target mind map. For example, the predicted root node is vibrated based on a certain frequency to achieve dynamic display of the predicted root node. The embodiment of the present application does not limit the specific display method of the visual display of the predicted root node, which is explained here.

[0145] ② Display the predicted root node in the target image in the form of a label in the target image; the label is used to indicate the predicted root node in the target mind map; in other words, the label displayed in the target image indicates the position of the predicted root node in the target image. The display style of the label varies depending on the display position of the label in the target image. For example, the label can be displayed as a highlighted red (or other color) dot in the area where the predicted root node is located (such as Figure 7b As another example, the annotation mark can be displayed in the target image in the form of annotation, such as Figure 7c As shown, the annotation mark may include: a comment box extending from the root node, in which the node at the end point of the extension line may be marked as the predicted root node.

[0146] It is understandable that the embodiments of the present application do not limit the specific display method of displaying the predicted root node in the target image; the above are just a few exemplary display methods given in the embodiments of the present application, which are specifically described here.

[0147] S303: In response to the confirmation operation on the predicted root node, output the target mind map.

[0148] As described above, after the computer device uses an algorithm to locate the predicted root node in the target mind map, the predicted root node will be displayed in the target image; by displaying the predicted root node, it is not only convenient for the user object to intuitively see the position of the located predicted root node, but also provides the user object with the confirmation authority for the located predicted root node. In other words, after the algorithm is used to locate the predicted root node of the target mind map, the embodiment of the present application also supports the user object to confirm the predicted predicted root node. Only when the user object confirms that the predicted predicted root node is the correct root node in the target mind map, the subsequent steps of restoring the entire target mind map will be continued, which can ensure that the restored target mind map is obtained based on the correct predicted root node restoration process, thereby improving the accuracy of the restored target mind map, and to a certain extent, can achieve efficient and high-quality restoration of the target mind map.

[0149] In a specific implementation, in addition to marking and displaying the located predicted root node in the image preview interface, the image preview interface also includes a confirmation option, so the user object can trigger the confirmation option in the image preview interface. At this time, it is determined that there is a confirmation operation for the predicted root node, which can trigger the execution of the steps for restoring the entire target mind map. Then, after the complete target mind map is restored, the complete target mind map can be output; the complete target mind map is in an editable state. Through the embodiment of the present application, it is possible to convert the target mind map in a non-editable state in the target image into a target mind map in an editable state, so that the user object can perform editing operations on the target mind map to meet the user object's editing needs for the target mind map. An exemplary schematic diagram of outputting a target mind map by triggering a confirmation operation in the image preview interface can be seen. Figure 8 ,like Figure 8 The image preview interface shown includes a confirmation option 801; in response to triggering the confirmation option 801, it indicates that the user object determines that the predicted root node is the correct root node in the target mind map, and the computer device calls the algorithm to continue to restore the entire target mind map based on the predicted root node, and outputs the restored target mind map.

[0150] In an embodiment of the present application, the computer device, in response to the restoration operation on the target mind map, can predict the root node of the target mind map in the target image and display the predicted root node of the target mind map in the target image; this can achieve automatic prediction of the root node in the target mind map, and compared with manual annotation of the root node, it avoids manual intervention in the confirmation process of the root node in the mind map, thereby reducing the workload of the user object. Furthermore, the computer device, in response to the user object's confirmation operation on the predicted root node, can continue to restore the complete target mind map based on the predicted root node; this restoration of the target mind map based on the accurate predicted root node obtained by prediction improves the accuracy of the restored target mind map. In the above scheme, in the restoration process of the target mind map, the complete target mind map can be quickly and accurately restored without manual intervention, thereby improving the restoration efficiency of the target mind map.

[0151] Figure 9 A flowchart of a mind map recognition method provided by an exemplary embodiment of the present application is shown; Figure 9 The mind map recognition method shown mainly involves the model application part and can be executed by a computer device (such as the computer device 201). The mind map recognition method may include but is not limited to steps S901-S904:

[0152] S901: Acquire a target image including a target mind map.

[0153] S902: In response to the restoration operation on the target mind map, calling the target semantic segmentation model to perform root node prediction processing on the target image, and displaying the predicted root node in the target mind map in the target image.

[0154] In steps S901-S902, the specific implementation process of obtaining the target image containing the target mind map and the specific implementation process of the restoration operation for the target mind map can be found in Figure 3 The relevant description of the specific implementation process shown in steps S301-S302 in the illustrated embodiment is not repeated here.

[0155] As described above, the restoration operation for the target mind map may include: the user object triggers the completion option in the image acquisition interface, then the computer device responds to the restoration operation for the target mind map, and can call the trained target semantic segmentation model to perform root node prediction processing on the target image, obtain the predicted root node in the target mind map, and then display the predicted root node in the target image to prompt the user object to confirm the predicted root node predicted by the computer device.

[0156] For ease of understanding, the following Figure 10 The specific implementation process of predicting the predicted root node in the target mind map in response to the restoration operation on the target mind map by the computer device is described in detail, which may include steps s11-s12:

[0157] s11: Call the target semantic segmentation model to perform root node prediction on the target image to obtain the initial predicted root node in the target mind map. Figure 10 As shown, first, the target image containing the target mind map is input into the trained target semantic segmentation model, so that the trained target semantic segmentation model performs semantic segmentation processing on the target image to obtain a target semantic label image (or score map) corresponding to the target image. The target semantic label image includes a foreground area and a background area. Among them, the target semantic label image is a grayscale image (that is, the thumbnails of the three channels of red (R), green (G), and blue (B) are all displayed in grayscale). The foreground area in the target semantic label image may include: the area where the adjacent pixels whose pixel values are not 0 are located. The foreground area is used to represent the area where the root node in the target mind map identified by the target semantic segmentation model is located; the background area in the semantic label image may include: the area where the adjacent pixels whose pixel values are 0 are located. The background area often refers to other areas in the target image except the foreground area.

[0158] Then, the target connected domain (or target connected domain) of the foreground area in the target semantic label image is calculated, and the target connected domain of the foreground area is used as the salient area in the target image, that is, the target connected domain is used as the area of interest to the user object in the target image. The target connected domain mentioned above may include the maximum connected domain of the foreground area; the maximum connected domain of the foreground area may refer to: the largest area in the foreground area composed of adjacent pixels with the same pixel value. Figure 10 As shown, the maximum connected domain in the foreground area is calculated and used as the salient area, which is displayed as a white area. By determining the salient area, the root node in the target mind map can be quickly locked to its approximate range in the target image, so that the root node can be found more accurately from the salient area later.

[0159] Finally, the center point of the salient area is used as the initial predicted root node in the target mind map. It is not difficult to understand that the salient area in the target image is often an irregular closed area; in order to facilitate finding the center point of the irregular closed area, the embodiment of the present application provides an optional method of finding the center point of the salient area; wherein, the process of finding the center point of the salient area may include: first generating the outline of the salient area, and then calculating the minimum circumscribed rectangle of the outline. The so-called minimum circumscribed rectangle or minimum boundary rectangle refers to the minimum rectangle that can completely include the salient area in the area, and finally calculating the center point of the minimum circumscribed rectangle, and using the center point as the initial predicted root node in the target mind map. Figure 10 As shown, the white area in the target semantic label image is the salient area, and the minimum enclosing rectangle of the salient area is a circumscribed square. Then, the intersection of the two diagonals of the circumscribed square can be determined as the center point of the circumscribed square, and then the center point of the circumscribed square can be used as the initial predicted root node in the target mind map. Specifically, the position of the center point is used as the position of the initial predicted root node.

[0160] Based on the above description, it can be seen that the salient area in the target semantic label image is used to represent the area of interest to the user object, and the area of interest is the area where the root node in the target mind map is located. Therefore, when the center point of the salient area is used as the initial predicted root node in the target mind map, the initial predicted root node can only be used to indicate: the approximate position of the root node in the target mind map in the target image. In order to obtain the accurate position of the root node in the target mind map, the embodiment of the present application also supports executing step s12 based on the initial predicted root node to determine the accurately predicted root node in the target mind map.

[0161] s12: Based on the initial predicted root node and each node in the target mind map, determine the node distance between the initial predicted root node and each node; then, determine the node with the shortest node distance to the initial predicted root node as the predicted root node. Specifically, the target image containing the target mind map can be input into a node detection model. The node detection model can be a network model for equivalently identifying all nodes of the target mind map in the target image. The so-called equivalent identification can be simply understood as not distinguishing the hierarchical relationship between each node (such as a root node or a child node) in the target mind map, but identifying all nodes. Then, calculate the node distance between the initial predicted root node located based on step s11 and each node identified by the node detection model. Finally, determine the node with the smallest node distance as the predicted root node in the target mind map.

[0162] The node distance between the initial prediction root node and any node may refer to the Euclidean distance, that is, the straight-line distance between the initial prediction root node and the any node. Figure 11 As shown in the figure, taking the upper left corner of the target image as the coordinate origin to establish a coordinate system, the coordinate information of the initial predicted root node and the coordinate information of each node in the target mind map can be obtained. Then, based on the coordinate information of the initial predicted root node and the coordinate information of each node, the Euclidean distance between the initial predicted root node and each node is calculated. For example, if the Euclidean distance between the initial predicted root node and node 1 is 0.2 cm, the Euclidean distance between the initial predicted root node and node 2 is 1 cm, the Euclidean distance between the initial predicted root node and node 3 is 1 cm, the Euclidean distance between the initial predicted root node and node 4 is 1.2 cm, and so on, it can be determined that the Euclidean distance between the initial predicted root node and node 1 is the shortest, and node 1 can be determined as the predicted root node in the target mind map.

[0163] In summary, the target semantic segmentation model is used to predict the initial predicted root node in the target mind map to indicate the approximate location of the root node in the target mind map; the Euclidean distance between the initial predicted root node and each node is then calculated, and the predicted root node of the target mind map can be accurately determined from all the nodes contained in the target mind map, and then the restoration process is subsequently performed with the accurate predicted root node as the starting point. Through the embodiment of the present application, the predicted root node in the target mind map can be automatically located, so that the entire target mind map restoration process is automatically and intelligently executed, thereby improving the restoration efficiency of the target mind map.

[0164] S903: In response to the confirmation operation on the predicted root node, output the target mind map.

[0165] It should be noted that the specific implementation process shown in step S903 can be found in Figure 3The relevant description of the specific implementation process shown in step S303 in the illustrated embodiment is not repeated here.

[0166] S904: In response to the operation of reselecting the root node in the target image, displaying the reselected new root node in the target image.

[0167] As described above, after the computer device has finished predicting the predicted root node in the target mind map, it can display the predicted root node in the target image so that the user object can judge whether the root node predicted by the computer device is accurate. If the user object recognizes the predicted root node predicted by the computer device (specifically the algorithm), that is, the user object confirms that the predicted root node is indeed the root node of the target mind map, then the user object can trigger the completion option in the image acquisition interface, and at this time it is determined that there is a confirmation operation for the predicted root node; the computer device can continue to perform subsequent steps to restore the entire target mind map based on the predicted root node. If the user object does not recognize the predicted root node predicted by the computer device, that is, the user object believes that the position of the predicted root node predicted by the computer device is wrong, then the embodiment of the present application also supports the user object to reselect the root node in the target image, so that the computer device can perform subsequent restoration steps based on the new root node reselected by the user object. Through the above implementation process, regardless of whether the predicted root node identified by the computer device is wrong, when the computer device subsequently restores the entire target mind map, it is based on the correct root node to perform the restoration process, which can ensure the accuracy of the restored target mind map.

[0168] In a specific implementation, when the user object determines that the predicted root node is incorrect, the user object can reselect the root node in the target image. At this time, the computer device responds to the operation of reselecting the root node in the target image by canceling the display of the predicted root node in the target image, specifically canceling the display of the annotation of the predicted root node, and displays the reselected new root node in the target image, specifically annotating and displaying the reselected new root node in the target image; wherein, the annotation display method of the reselected new root node can be found in the aforementioned description, and the relevant description of the annotation display method of the predicted root node will not be repeated here. An exemplary schematic diagram of an exemplary reselection of a new root node can be found in Figure 12 ;like Figure 12As shown, after the computer device predicts the predicted root node of the target mind map, the predicted root node can be marked and displayed in the image preview interface 1201, and a prompt message 1202 can be displayed. The prompt message 1202 is used to prompt the user object: if the predicted root node is incorrect, the user object can reselect a new root node. Further, if the user object determines that the predicted root node is incorrect, the select starting point option 1203 in the image preview interface 1201 can be triggered. At this time, the mark display of the predicted root node can be deleted in the target image, and the select starting point option 1203 can be highlighted to prompt the user object to start the operation of reselecting the root node. Then, the user object can select (such as single-click) any node in the target image. In response to the user object's selection operation on any node in the target image, the computer device determines the selected node as the reselected new root node and marks and displays the selected node in the target image.

[0169] The embodiment of the present application also supports canceling the user object's click operation on the node; this can meet the user object's need to select the root node multiple times, improve the flexibility of the root node selection, and thus enhance the user object's experience. Figure 12 ,like Figure 12 The image preview interface shown also includes an undo option 1204; when the user object triggers the undo option 1204, it means that the user object wants to cancel the most recent click operation at the current moment (i.e., the moment when the undo option is triggered), and the label display of the most recently clicked node is canceled in the target image, and the label displays the root node determined before the most recently clicked node, which may be a predicted root node or a root node selected by the user.

[0170] In addition, considering that the display area of the terminal screen of the computer device is limited, when the target mind map has many nodes or contents, the target mind map displayed on the terminal screen will not be clear; based on this, the embodiment of the present application also supports the user object to adjust the target image in the image preview interface, so that the target mind map in the adjusted target image can be displayed more clearly, which can meet the needs of the user object to more clearly browse the various parts of the target mind map contained in the target image. Among them, the adjustment of the target image may include: adjusting the display area size of the target image in the image preview interface, and / or adjusting the display part of the target image in the image preview interface (such as the partial image of the target image displayed on the terminal screen); and the embodiment of the present application does not limit the execution order of these two adjustments, such as first adjusting the display area of the target image in the image preview interface, and then adjusting the display part of the target image in the image preview interface. The following are several exemplary implementation processes of user objects adjusting the target image in the image preview interface, among which:

[0171] Optionally, by performing a gesture operation in the image preview interface, the display area size of the target image displayed in the image preview interface can be adjusted, thereby adjusting the display area of the target mind map contained in the target image. Figure 13a As shown, the computer device responds to the two-finger pinch operation in the image preview interface, and according to the two-finger pinch operation, reduces the display area of the target image in the image preview interface until the display area of the target image is the same as the display area of the area used to display the target image in the image preview interface; as the display area of the target image is enlarged, the content contained in the target image - the target mind map, is also enlarged, so that the user object can browse the content contained in the target mind map more clearly. Similarly, if Figure 13b As shown, in response to a two-finger pinch operation in the image preview interface, the computer device can enlarge the display area of the target image in the image preview interface according to the two-finger pinch operation until the display area of the target image reaches a preset display area threshold. Of course, the two-finger pinch operation or two-finger pinch operation described above are merely exemplary gesture operations for adjusting the target image; embodiments of the present application do not limit this.

[0172] Optionally, the display area of the target image can be adjusted through controls (or components, options, keys, buttons, etc.), thereby adjusting the display area of the target mind map contained in the target image. Figure 13c As shown, the image preview interface also includes a zoom-in control 1301 and a zoom-out control 1302. When either control is triggered, it indicates that the user object wants to perform the function corresponding to the triggered control on the target image. For example, in response to the user object triggering the zoom-in control 1301, the target image can be zoomed in on the image preview interface. Each time the zoom-in control (or zoom-out control) is triggered, the corresponding zoom size of the target image can be pre-configured by the business personnel. For example, each time the zoom-in control is triggered, the display area of the target image in the image preview interface can be magnified by 1.1 times. The specific zoom size is not limited in this embodiment of the application and is explained here.

[0173] Optionally, by sliding the target image in the image preview interface, the displayed portion of the target image can be adjusted, thereby adjusting the displayed portion of the target mind map. Figure 13d As shown, if the computer device detects a sliding operation to the right in the image preview interface, the target image can be slid to the right according to the sliding operation, so that the left part of the target image is visible and the right part is hidden; in this way, it can be ensured that when the display area of the target image is large, the user object can view any part of the content, thereby improving the user object's usage experience.

[0174] It should be noted that the above examples only provide several exemplary implementations of adjusting the target image in the image preview interface; however, depending on the computer device, the implementation methods for adjusting the target image in different application scenarios may vary. For example, if the computer device uses an external mouse to adjust the target image, the display area of the target image can be zoomed in and out by scrolling the mouse; the specific adjustment method for adjusting the target image is not limited in the embodiments of this application.

[0175] S905: In response to the confirmation operation on the new root node, output the target mind map.

[0176] The specific implementation process of the confirmation operation of the new root node described in step S904 is the same as the above Figure 3 The specific implementation process of the confirmation operation on the predicted root node described in the illustrated embodiment is similar and will not be described in detail here.

[0177] In a specific implementation, in response to the confirmation operation of the new root node, the computer device can use a depth-first traversal method, with the new root node as the starting point, to find the child nodes in the target mind map that are related to the starting point, and advance layer by layer to finally restore the complete structure of the mind map. Among them, the process of restoring the target mind map using the depth-first traversal method can be simply summarized as follows: starting from the root node (such as the predicted root node or the new root node selected by the user object), find a path and continue to explore the nodes on the path in depth; when it is detected that the path is blocked, it will fall back to the last explored node; if there is no branch to be explored at the previous node, it will continue to explore, and if there is no branch to be explored, it will continue to fall back until every path in the target mind map is traversed. Among them, the path in the target mind map may refer to: starting from the root node, using mind map lines to connect the root node with one or more child nodes.

[0178] After the computer device restores the target mind map to obtain a complete target mind map, the target mind map can be output using the editing interface of the mind map tool (or mind map service) provided by the target application. It should be noted that the target mind map is displayed in the editing interface of the mind map tool, and the editing interface can be understood as a service interface for editing (such as creating) a mind map in the mind map tool; therefore, the embodiment of the present application supports the user object to edit the displayed target mind map in the editing interface, such as using the functions provided by the mind map tool to modify the node content in the target mind map in the editing interface.

[0179] Furthermore, the embodiment of the present application supports the user object to perform editing operations on the target mind map displayed in the editing interface. Specifically, the computer device updates the target mind map in response to the editing operation on the target mind map in the editing interface; the editing operation specifically includes the operation of editing the target mind map using the functions provided by the mind mapping tool; the editing operation may include but is not limited to: modifying the color, style, and content of the existing nodes in the target mind map (such as copy and paste operations), modifying the thickness, style, and color of the mind map lines in the target mind map, adding nodes to the target mind map, and deleting existing nodes in the target mind map. One or more of the following. For example: if the mind map lines in the target mind map are solid lines, then the editing operations performed on the target mind map may include: modifying the mind map lines of the entities in the target mind map to dotted lines. For another example: if the root node in the target mind map is represented by a white circular box, then the editing operations performed on the target mind map may include: modifying the white circular box to a rectangular red box; and so on. The user object can perform editing operations on the target mind map in the editing interface according to its own business needs, and the embodiment of the present application will not be described in detail here.

[0180] The following combination Figure 14 , and taking the editing operation including modifying the content contained in the node as an example, a schematic diagram of an exemplary editing operation is given; Figure 14 As shown, if the root node in the editing interface 1401 is triggered, the root node is displayed in an editable state. The display style of the root node in the editable state is different from the display style of other nodes, such as Figure 14 The root node shown is displayed in the editing box 1402; and an editing bar 1403 is output in the editing interface 1401, and the editing bar 1403 includes one or more editing options, such as an edit item, a paste item, etc. After any editing option is triggered, the function corresponding to the triggered editing option can be executed. If an editing item is triggered, a toolbar 1404 is output in the editing interface 1401, and the toolbar 1404 includes one or more tools, such as a keyboard, a left branch, a right branch, a font, etc.; if the keyboard tool is triggered, a virtual keyboard is displayed in the editing interface, so that the user object can re-edit the content contained in the root node through the virtual keyboard, and after the editing is completed, the new content obtained by the editing is used to update the original content contained in the root node, so as to realize the editing of the content contained in the root node.

[0181] Furthermore, the embodiment of the present application also supports sharing the edited target mind map. In a specific implementation, the computer device generates a sharing image based on the edited target mind map in response to the editing completion operation in the editing interface, and the sharing image includes the edited target mind map; then, the sharing image is shared to the sharing object to realize the sharing of the edited target mind map. Depending on the mind mapping tool, the editing completion operation performed in the editing interface provided by the mind mapping tool is different; the following is Figure 15 Taking the editing interface shown in FIG. 1 as an example, a schematic diagram of sharing the edited target mind map is given; Figure 15 As shown, the editing interface includes a share option 1501. When the share option 1501 is triggered, it indicates that the user wants to share the target mind map currently displayed in the editing interface. A share image can be generated based on the currently displayed target mind map, and a list of share objects 1502 can be displayed. The list of share objects may include: the identifier of the share object, a link to the share image, etc.

[0182] Optionally, the sharing object may include a recipient of the shared image, in which case the identifier of the sharing object includes information that can be used to uniquely identify the identity of the sharing object (such as an account number); then, in response to the user object triggering the identifier of any sharing object, the shared image can be shared with any sharing object. Optionally, the sharing object may also include a service provided by an application other than the target application, such as a social dynamics service provided by a social application, in which case the identifier of the sharing object includes the service identifier of the social dynamics service; then, in response to the user object's service identifier of the social dynamics service, the shared image can be shared to a social dynamics interface provided by the social application, which includes a dynamic message stream (or feed stream) consisting of dynamic messages sent by one or more user objects. The dynamic message stream can be dynamically updated in the social dynamics interface by refreshing. Optionally, if a shared image link in the sharing list is selected, the user object can share the shared image link of the shared image, so that the recipient can obtain the shared image through the shared image link. The embodiments of the present application do not limit the specific sharing process of sharing images. The above are just a few exemplary sharing implementation methods, which are explained here.

[0183] In an embodiment of the present application, in response to a restoration operation on a target mind map, the computer device can call a trained target semantic segmentation model to predict the root node of the target mind map in the target image, thereby automatically predicting the predicted root node in the target mind map without manual intervention, thereby reducing the workload of the user object. In addition, the embodiment of the present application also supports the user object to re-select a new root node in the target image when it is determined that the predicted root node is incorrect, so that the computer device can restore the target mind map based on the new root node, thereby ensuring the accuracy of the target mind map finally restored. In addition, the embodiment of the present application also supports editing and sharing of the restored target mind map, meeting the user object's needs for editing and sharing the target mind map, and improving the user object's usage experience.

[0184] above Figure 3 and Figure 9 The illustrated embodiment mainly provides a specific implementation process of the model application, and the following is an introduction to the relevant content of the model training part involved in the embodiment of the present application. It should be noted that the semantic segmentation model provided in the embodiment of the present application is used to predict the semantic segmentation image corresponding to the input image, and the semantic segmentation image includes the foreground area where the root node of the mind map is located; that is, the goal of training the initial semantic segmentation model in the embodiment of the present application is to enable the trained target semantic segmentation model to predict the area where the root node is located. In this way, the correct root node in the mind map can be identified based on the foreground area in the semantic segmentation image. In more detail, the embodiment of the present application uses supervised learning to train the initial semantic segmentation model. The so-called supervised learning can be understood as: using a training data set to train the initial semantic segmentation model, and the obtained trained target semantic segmentation model can make a relatively accurate prediction of the area where the root node is located in other images other than the training data set. The overall idea of the embodiment of the present application to train the initial semantic segmentation model using supervised learning is: first determine the correct semantic segmentation image corresponding to the training image, and then use the correct semantic segmentation image to train the initial semantic segmentation model; so that the trained target semantic segmentation model can identify the area where the root node of the target mind map included in any target image is located; and then subsequently, the target semantic segmentation image corresponding to the target image predicted by the target semantic segmentation model can accurately obtain the predicted root node. Among them, the specific implementation process of accurately identifying the predicted root node in the target mind map based on the target semantic segmentation image can be found in the aforementioned Figure 9 The relevant description of the specific implementation process shown in step s11 of step S902 in the illustrated embodiment is not repeated here.

[0185] The following combination Figure 16, taking the training process of the initial semantic segmentation model using a training image as an example, the content of the model training part provided in the embodiment of this application is elaborated in detail. Figure 16 A flowchart of a mind map recognition method provided by an exemplary embodiment of the present application is shown; the mind map recognition method can be executed by a computer device (such as the computer device 202), and the mind map recognition method may include but is not limited to steps S1601-S1607:

[0186] S1601: Acquire a training image.

[0187] In a specific implementation, training images can be obtained from a training image set; the training image set includes a plurality of sample images for training the initial semantic segmentation model, and the training image is any sample image in the training image set. The sample images included in the training image set can be divided into a first sample image and a second sample image; the first sample image can be a sample image directly synchronized from the Internet and / or local storage space and containing a training mind map; the second sample image is a sample image obtained by performing data enhancement processing (or data expansion processing) on the first sample image; then one or more first sample images and one or more second sample images are merged to obtain a training image set. It should be noted that the sample images in the training image set may also include only the first sample image, and the embodiment of the present application does not limit the specific method of obtaining the sample images in the training image set.

[0188] Among them, the data enhancement processing performed on the first sample image may include but is not limited to: one or more of blurring, scaling, mirroring, rotation, cropping or noise enhancement processing. For example, blurring can also be called filtering, which mainly makes the first sample image blurry and extracts important information of the first sample image; blurring includes but is not limited to: Gaussian blur, median blur, mean (salt and pepper) blur, etc. For another example: scaling may refer to achieving data enhancement of the first sample image by changing the display area of the first sample image. For another example, rotation may refer to keeping the display area of the first sample image unchanged, rotating the first sample image up and down and / or left and right, etc. to increase the amount of data. The embodiment of the present application does not limit the specific type of data enhancement processing, which is explained here.

[0189] Furthermore, after obtaining the training image set, the embodiment of the present application also supports labeling the sample images in the training image set, specifically marking the location of the root node in the training mind map in each sample image, so as to facilitate the subsequent use of the sample image with the location of the root node marked to train the initial semantic segmentation model to be optimized. In a specific implementation, a root node labeling system can be used to label the location of the root node in the training mind map included in each sample image. The root node labeling system can refer to a labeling system that supports business personnel (or labeling personnel) to manually label the root nodes of the mind map. The specific implementation process of using a root node annotation system to annotate the location of a root node may include: in the root node annotation system, each time presenting an unannotated sample image to the annotator, so that the annotator can trigger (such as clicking) the location of the root node in the sample image; then, the root node annotation system, in response to the annotator's trigger in the interface, can automatically record the coordinate information of the location triggered by the annotator in the interface, and use the coordinate information as the coordinate information of the root node in the sample image; save the coordinate information to a text document with the same name as the sample image to achieve the record of the location of the root node in the sample image. Then, the root node annotation system can automatically switch to the next unannotated training image to facilitate the annotator to continue the root node annotation operation on the next unannotated training image.

[0190] For example: Assume that the sample image is Figure 1a As shown, the root node area in the training mind map included in the sample image is an image block (or text block) containing text. When it is detected that the annotator triggers any position in the image block, the center position of the image block is determined as the coordinate position of the root node; and the coordinate information of the center position is stored in a text document with the same file name as the sample image to achieve the root node annotation of the sample image. For another example: Assume that the sample image is as follows Figure 1b As shown, the root node in the training mind map included in the sample image is the endpoint of a line. When it is detected that the annotation personnel triggers the endpoint, the coordinate information of the endpoint is determined as the coordinate information of the root node; and the coordinate information of the endpoint is stored in a text document with the same file name as the sample image to realize the annotation of the root node of the sample image.

[0191] Through the above implementation process, the root node of each sample image in the training image set can be annotated, and a root node annotation file corresponding to each sample image can be obtained. In other words, the dataset used to train the initial semantic segmentation model to be optimized includes not only the training image set but also a set of root node annotation files corresponding to the training image set. The root node annotation file set includes a root node annotation file corresponding to each sample image in the training image set. Each root node annotation file records the coordinate information of the root node in the sample image corresponding to the root node annotation file.

[0192] S1602: Perform a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image.

[0193] As described above, the embodiment of the present application adopts a supervised learning approach to train the initial semantic segmentation model; therefore, before training the initial semantic segmentation model, it is necessary to first determine the first semantic label image used to train the initial semantic segmentation model, and the foreground area included in the first semantic label image is the area where the root node in the training mind map in the training image is located. Among them, the implementation process of performing the first semantic segmentation processing on the training mind map in the training image to obtain the first semantic label image can be simply described as: first, extract the salient area in the training image to obtain the attention image (or attention heat map) corresponding to the training image; then, perform feature extraction on the attention image to obtain a feature map; then, combine the attention image and the feature map to redefine the labels of each feature point in the feature map to obtain the first semantic label image. The following is combined with Figure 17 A specific implementation process of determining the first semantic label image corresponding to the training image is given. The implementation process may include but is not limited to steps s21-s23:

[0194] s21: Perform region extraction on the training image to obtain the attention image.

[0195] From the above description, it can be seen that the training image corresponds to a root node annotation file, and the root node annotation file includes the coordinate information of the root node in the preset training mind map; the coordinate information of the root node in the preset training mind map here is obtained by the annotation personnel using the root node annotation system to annotate in the training image. Then, the root node annotation file corresponding to the training image can be obtained, and then, based on the coordinate information in the root node annotation file, the training image is subjected to region extraction processing (or called salient region extraction processing) to generate an attention image. Among them, the attention image includes: a salient region centered on the coordinate position indicated by the coordinate information; such as Figure 17 The color depth of the salient regions in the attention image is shown to be greater than that of other regions in the attention image.

[0196] Specifically, the salient region extraction rules for performing region extraction processing on the training image may include: taking the coordinate position indicated by the coordinate information in the root node annotation file as the center point (or called the attention center), radiating to the surrounding areas to generate an attention image. In this implementation, the salient region in the generated attention image refers to: a circular region centered on the coordinate position indicated by the coordinate information, and the color depth in the salient region is greater than the color depth of other regions in the attention image; the darker the color, the higher the user object's attention to the region. In a specific implementation, considering that the radius radiating from the attention center to the surrounding areas is related to the size of the training image, the display size information of the training image is first obtained, and the display size information includes the width information and height information of the training image; then the radiation radius of the salient region can be calculated based on the display size information; wherein, the calculation formula for the user to calculate the radiation radius is as follows:

[0197] r=min(w,h)×RADIUS_RARIO Formula 1

[0198] r is the radius of the salient region, w is the width of the training image, h is the height of the training image, min(w,h) is the minimum of the width w and the height h, and RADIUS_RATIO is an empirical value. The specific value of RADIUS_RATIO can be adjusted based on the characteristics of the actual training image used; for example, in this embodiment of the application, a RADIUS_RATIO value of 0.25 results in a more appropriate radius of the salient region.

[0199] Then, after the radiation radius of the salient area is calculated based on the above formula, the salient area in the training image can be determined according to the coordinate information and radiation radius in the root node annotation file. Figure 16 The salient area determined in the attention image shown includes: a circular area with the coordinate position indicated by the coordinate information in the training image as the center point and the radiation radius r as the radius.

[0200] Finally, the thermal value of each feature point in the salient area is calculated to generate an attention image. The embodiment of the present application uses thermal values to represent the user object's attention to the pixel point (or feature point); the higher the thermal value of the feature point in the attention image, the higher the user object's attention to the feature point, and the greater the probability that the feature point is the feature point where the root node is located. It can be understood that the attention level of the position where the attention center is located in the salient area is the highest, and the thermal value of the feature point at the position where the center is located is the highest, and then the thermal value of each feature point gradually decays to the surrounding area according to the radiation radius r; wherein, the calculation formula for calculating the thermal value of each feature point in the salient area is as follows:

[0201]

[0202] Heatmap is an attention image (or attention heatmap), where heatmap(j,i) is the heat value of the pixel (or feature point) in the jth row and ith column of the attention image, and (x,y) is the attention center marked in the attention image. The calculation formula shows that the maximum heat value of a feature point in the attention image is 1, and the minimum heat value is 0. The larger the heat value of a feature point in the attention image, the higher the attention value of the feature point, and the greater the probability that the feature point is a feature point in the region where the root node is located.

[0203] Based on the above implementation process, the training image can be processed for salient area extraction to generate an attention image; the attention image includes a salient area, the thermal value of each feature point in the salient area is greater than the thermal value threshold of 0, and the thermal value of the feature points in other areas of the attention image except the salient area is the thermal value threshold of 0.

[0204] It should be noted that the embodiment of the present application also supports setting the thermal value of each feature point in the salient area to the same thermal value; under this implementation, the thermal value of the feature point in the salient area in the attention image is a preset thermal value (such as 1), and the thermal value of the feature point in other areas of the attention image except the salient area is 0. Alternatively, the embodiment of the present application also supports the use of a prediction model to predict the training image to obtain the salient area in the training image. The embodiment of the present application does not limit the specific implementation process of the salient area extraction rule for extracting the salient area in the training image. According to actual application requirements, other rules with the same function can be used to replace the salient area extraction rule mentioned above, which is specially explained here.

[0205] s22: Feature extraction is performed on the attention image to obtain a feature map. The purpose of feature extraction is to map the high-dimensional feature space of the attention image to a low-dimensional feature space. Each pixel in the feature map after feature extraction is assigned a category label, and the label value of the category label is the pixel value of the corresponding pixel; and the pixel values of adjacent pixels of the same object in the feature map are the same, so that when the feature map is visualized, the same object in the feature map (such as a car, a human body or a table, etc.) appears as the same color, which facilitates the classification and identification of each object in the feature map. The embodiment of the present application does not limit which feature extraction algorithm is used to extract features from the attention image; for example, a neural network with a feature extraction function can be used to extract features from the attention image to obtain a feature map corresponding to the attention image.

[0206] S23: Redefine the category labels in the feature map based on the attention image to obtain a first semantic label image. The process of redefining the category labels may include replacing the category label values of the corresponding feature points in the feature map with the thermal values of the feature points in the salient region of the attention image, thereby redefining the pixel values of the feature points in the feature map.

[0207] For example, assuming that the thermal value of a target feature point (such as any feature point) in a salient area of an attention image is 0.5, and the label value of the category label of the target feature point in the feature map is 156, then the redefinition process may include: assigning the thermal value of the target feature point in the attention image to the category label of the target feature point in the feature map, so that the label value of the category label of the target feature point in the feature map is 0.5. Through the above-mentioned process of redefining the category label, it is possible to separate the salient area (such as a thermal value greater than 0, and the closer to the center position, the greater the thermal value) from other areas (such as a thermal value of 0), and obtain a first semantic label image including a foreground area and a background area. The pixel values of the feature points in the foreground area of the first semantic label image are not 0, so that the visual effect of the foreground area is a circular area with the center of attention as white and the grayscale gradually deepening; similarly, the pixel points in the background area of the first semantic label image are 0, so that the visual effect of the background area is black.

[0208] The aforementioned process of extracting salient regions, extracting features, and then redefining labels on a training image yields a true first semantically labeled image for the training image. This image can then be used to supervise the training of the optimized initial semantic segmentation model. The trained target semantic segmentation model can accurately predict the semantically segmented image for any input image.

[0209] S1603: Calling the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantic label image.

[0210] The semantic segmentation model is introduced below. The semantic segmentation model provided in the embodiment of the present application can also be called a semantic segmentation network or a segmentation network. It is a network model that implements semantic segmentation of images. The so-called semantic segmentation refers to a deep learning algorithm that associates a label or category with each pixel of an image, which can be used to identify the set of pixels that constitute the distinguishing category. A simple example of semantic segmentation is to divide an image into two categories, such as Figure 18aThe image shown includes a penguin swimming. Semantic segmentation of the image can obtain two different categories of image pixels, namely, penguin and background. Similar to the above example, semantic segmentation is also applied to the embodiment of the present application in a binary classification manner, that is, the semantic label image obtained by semantic segmentation of the training image can include two different categories of image pixels, such as Figure 18b As shown, the semantic label image includes: a root node area (ie, the foreground area mentioned above) and a background area (ie, the area other than the foreground area in the training image).

[0211] An exemplary model structure of the semantic segmentation model provided in the embodiment of the present application can be found in Figure 19 ,like Figure 19 As shown, the semantic segmentation model mainly includes: a downsampling module (stemnet), an N-layer feature extraction module (such as 4stageconv), a feature fusion module (feature concat) and a classification module (segmenthead). Among them: ① Downsampling module (stemnet), including 2 3*3 convolutions with a stride of stride=2, so that the training image input to the initial semantic segmentation model can be downsampled (or called downsampling) to 1 / 4 of the original resolution (that is, the resolution before the training image is input to the initial semantic segmentation model). Of course, according to different business needs, the convolution kernel and stride of the convolution contained in the downsampling module can change, and the embodiment of the present application does not limit this. ② N-layer feature extraction module (such as 4stageconv), N is an integer greater than 1; taking N=4 as an example, the first stage includes 4 residual units, each residual unit contains 2 3*3 convolutions, the second, third, and fourth stages contain 1, 4, and 3 multi-resolution blocks respectively, and each branch in the multi-resolution group convolution contains 4 residual units. Through the 4-stageconv, feature maps of different resolutions can be connected in parallel, with the same resolution occupying one branch and different resolutions occupying different branches, and adding paths between different branches (i.e. Figure 19 ), forming a high-resolution network; the high-resolution network can realize the feature acquisition of different resolutions of the downsampled image, enriching the features acquired by the model. ③ Feature fusion module (feature concat), which can be used to upsample the feature map corresponding to the lower resolution layer to high resolution, and fuse the multiple feature maps with the same resolution after upsampling to obtain a new feature map. The lower resolution layer here can refer to Figure 19The three layers with lower resolution shown; that is, the feature map corresponding to each of the three layers with lower resolution can be upsampled to the same resolution as the feature map corresponding to the layer with the highest resolution, so as to unify the resolutions of multiple feature maps. Among them, the upsampling method may include but is not limited to: linear interpolation (such as bilinear interpolation) or deconvolution, and the embodiment of the present application does not limit the upsampling method. ④ The classification module (segmenthead) can superimpose the features of multiple feature maps of the same resolution processed by the upsampling module, so as to predict the second semantic label image corresponding to the training image; specifically, the features of multiple feature maps of the same resolution processed by the upsampling module can be connected by 1*1 concat (such as superposition) to obtain a mixed representation; then, the mixed representation of each pixel is passed to the classification module, so that the classification module predicts the second semantic label image.

[0212] Based on the above introduction to the model structure of the semantic segmentation model, the following describes the implementation process of performing the second semantic segmentation processing on the training image using the initial semantic segmentation model. The specific implementation process may include:

[0213] First, after obtaining a training image, it can be preprocessed and the preprocessed training image of the target dimension used as the network input for the initial semantic segmentation model. This preprocessing process may include adjusting the size of the training image based on the short side of the training image. Preprocessing the training image can minimize computational redundancy caused by image compression or cropping, while also preserving the structural information of the image frame. When the width w of the training image is greater than or equal to the height h, indicating that the short side of the training image is the height h, the height h can be fixed to a preset value (e.g., 448). In this case, the dimensions of the preprocessed training image are: n*448*w*3. When the width w of the training image is less than the height h, indicating that the short side of the training image is the width w, the width w can be fixed to a preset value (e.g., 448). In this case, the dimensions of the preprocessed training image are: n*h*448*3. In the target dimension formula, n represents the number of training images in each batch input to the initial semantic segmentation model, and the value 3 represents the number of channels.

[0214] Next, the downsampling module (stemnet) included in the initial semantic segmentation model is called to downsample the preprocessed training image to obtain a first-resolution image. Then, the multi-layer feature extraction module (e.g., an N-order convolutional layer) in the initial semantic segmentation model is called to extract features from the first-resolution image. This allows for feature extraction of the first-resolution image at different resolutions, enriching the extracted image features and obtaining N feature maps of different resolutions.

[0215] Then, the feature fusion module (feature concat) in the initial semantic segmentation model is called to perform feature fusion processing on N feature maps to obtain the fused target feature map. The specific implementation process of feature fusion processing can be found in Figure 20 ,like Figure 20 As shown in FIG, first, the feature maps with a resolution less than a resolution threshold in the N feature maps are upsampled to obtain N feature maps with the same resolution, wherein the resolution threshold may refer to the resolution corresponding to the layer with the highest resolution in the multi-layer feature extraction module; then, the N feature maps with the same resolution are fused (or superimposed) to obtain a target feature map; at this time, each pixel point in the target feature map has a new representation, and the new representation of any pixel point is a mixed representation of the corresponding pixel points in the N feature maps.

[0216] Finally, the classification module (segment head) in the initial semantic segmentation model is called to classify the target feature map and obtain the second semantic label image. The classification module (segment head) can be simply understood as an activation function (such as the softmax function) that converts the output values of multiple classifications into a probability distribution between [0, 1] and 1. The softmax function is defined as follows:

[0217]

[0218] Among them, z k is the output value (e.g., pixel value) of the kth category, and C is the number of categories, i.e., the number of categories classified. In the embodiment of the present application, the number of categories classified is 2, i.e., the embodiment of the present application performs a binary classification process on the target feature map, including foreground and background categories; the foreground region and the background region are represented in the second semantic label image obtained by the classification process.

[0219] It should be noted that Figure 19 This is just a schematic diagram of the module structure of an exemplary semantic segmentation model; in practical applications, the same Figure 19 The semantic segmentation model shown can be replaced by other network structures with the same function, and the embodiment of the present application does not limit the specific structural style of the network structure.

[0220] S1604: Obtain a loss function of the initial semantic segmentation model, and calculate a loss value of the loss function based on the first semantic label image and the second semantic label image, so as to train the initial semantic segmentation model according to the loss value.

[0221] In the specific implementation, the loss function of the initial semantic segmentation model is used to calculate the loss value, and the initial semantic segmentation model is continuously optimized according to the loss value to obtain the trained target semantic segmentation model. Among them, the loss function (or optimization function) of the initial semantic segmentation model can be expressed as:

[0222] pixeloss=-∑y true log(y pred ) Formula 4

[0223] Among them, pixelloss is the loss function, y true is the real first semantic label image corresponding to the training image, y pred The predicted second semantic label image is obtained by using the initial semantic segmentation model to predict the training image.

[0224] Furthermore, if the calculated loss value of the loss function satisfies a preset condition, indicating that the initial semantic segmentation model has reached convergence, i.e., the initial semantic segmentation model has good prediction performance, then the initial semantic segmentation model obtained in this round of training can be determined as the trained target semantic segmentation model. The loss value satisfying the preset condition may include, but is not limited to, a loss value less than a loss threshold, or a difference between loss values obtained from several consecutive training runs less than a difference threshold (e.g., 0). It should be noted that the loss threshold may vary for different network models and is not detailed here. Conversely, if the calculated loss value does not meet the preset condition, indicating that the prediction performance of the initial semantic segmentation model does not meet the requirements, the initial semantic segmentation model is iteratively trained. Specifically, the initial semantic segmentation model can be optimized using a gradient descent method, such that the semantic label image predicted by the optimized initial semantic segmentation model increasingly approximates the true semantic label image. The implementation process of optimizing the initial semantic segmentation model using the gradient descent method can be simply understood as: optimizing the network parameters of the initial semantic segmentation model using the loss value, and continuing to train the optimized initial semantic segmentation model using other samples from the training image set if the number of training runs is less than the preset total number of runs.

[0225] Through the above implementation process, a trained target semantic segmentation model can be obtained; since the training mind maps contained in each training image in the training image set used to train the semantic segmentation model are different, this enables the trained target semantic segmentation model to predict the root nodes in various mind maps, so that the target semantic segmentation model has good generalization ability and can perform root node prediction processing on different mind maps.

[0226] S1605: Acquire a target image including a target mind map.

[0227] S1606: In response to the restoration operation on the target mind map, the target semantic segmentation model is called to perform root node prediction processing on the target image, and the predicted root node is output.

[0228] S1607: In response to the confirmation operation on the predicted root node, the target mind map is restored based on the predicted root node, and the restored target mind map is output.

[0229] It should be noted that the specific implementation process shown in steps S1605-S1607 can be found in the aforementioned Figure 3 and / or Figure 9 The relevant description of the specific implementation process in the illustrated embodiment is omitted here.

[0230] In an embodiment of the present application, support is provided for training an initial semantic segmentation model using training images comprising different training mind maps, enabling the trained target semantic segmentation model to predict root nodes in various mind maps, ensuring that the target semantic segmentation model has good generalization capabilities and can perform root node prediction processing on different mind maps. Furthermore, in response to a restore operation on a target mind map, the computer device can invoke the trained target semantic segmentation model to automatically predict the root node of the target mind map in the target image without manual intervention, thereby reducing the workload of the user object and improving the user object's user experience.

[0231] The above describes in detail the method of the embodiment of the present application. In order to facilitate better implementation of the above method of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0232] Figure 21 A schematic diagram of the structure of a mind map recognition device provided by an exemplary embodiment of the present application is shown. The mind map recognition device may be a computer program (including program code) running on a computer device; the mind map recognition device may be used to execute Figure 3 、 Figure 9 and Figure 16 Part or all of the steps in the method embodiment shown; the device includes the following units:

[0233] An acquiring unit 2101 is configured to acquire a target image including a target mind map;

[0234] The processing unit 2102 is configured to display the predicted root node in the target mind map in the target image in response to the restoration operation on the target mind map;

[0235] The processing unit 2102 is further configured to output a target mind map in response to a confirmation operation on the predicted root node, where the target mind map is in an editable state.

[0236] In one implementation, when the processing unit 2102 is configured to display the predicted root node in the target mind map in the target image, it is specifically configured to:

[0237] Annotate the target image with the predicted root node in the target mind map;

[0238] The annotation display includes: displaying the predicted root node in the form of a visual display; or displaying the predicted root node in the form of an annotation mark;

[0239] The annotation mark is displayed in the area where the predicted root node is located; or, the annotation mark is displayed in the target image in the form of an annotation.

[0240] In one implementation, when the processing unit 2102 is configured to obtain a target image including a target mind map, it is specifically configured to:

[0241] Display the function selection interface, which includes the option of converting images to mind maps;

[0242] In response to triggering the image-to-mind-map option, displaying an image acquisition interface;

[0243] In the image acquisition interface, a target image containing a target mind map is acquired.

[0244] In one implementation, the image acquisition interface includes an image upload option. When the processing unit 2102 is configured to acquire a target image including a target mind map in the image acquisition interface, the processing unit 2102 is specifically configured to:

[0245] In response to triggering the image upload option, displaying at least one candidate image;

[0246] Any candidate image is selected from at least one candidate image, the selected candidate image is used as a target image, and a mind map included in the candidate image is used as a target mind map included in the target image.

[0247] In one implementation, the image acquisition interface includes an image scanning option. When the processing unit 2102 is configured to acquire a target image including a target mind map in the image acquisition interface, the processing unit 2102 is specifically configured to:

[0248] In response to triggering the image scanning option, a scanning operation is performed on the target image containing the target mind map to obtain the target image.

[0249] In one implementation, the processing unit 2102 is further configured to:

[0250] In response to an operation of reselecting a root node in the target image, canceling the labeling display of the predicted root node in the target image, and labeling and displaying the reselected new root node in the target image;

[0251] In response to a confirmation operation on the new root node, outputting a target mind map;

[0252] The annotation display of the predicted root node is revoked to indicate that the predicted root node has not been confirmed.

[0253] In one implementation, the processing unit 2102 is further configured to:

[0254] In response to an editing operation on a target mind map, updating the target mind map;

[0255] In response to a sharing operation on the updated target mind map, a sharing process is performed on a sharing image including the updated target mind map.

[0256] In one implementation, the processing unit 2102 is configured to, in response to a restoration operation on a target mind map, display a predicted root node in the target mind map in a target image, specifically:

[0257] In response to the restoration operation on the target mind map, calling the target semantic segmentation model to perform root node prediction processing on the target image to obtain an initial predicted root node in the target mind map;

[0258] According to the initial prediction root node and each node in the target mind map, determining the node distance between the initial prediction root node and each node;

[0259] The node with the shortest distance to the initial predicted root node is used as the predicted root node.

[0260] In one implementation, the processing unit 2102 is configured to call the target semantic segmentation model to perform root node prediction processing on the target image, and obtain an initial predicted root node in the target mind map, specifically for:

[0261] Call the target semantic segmentation model to perform semantic segmentation processing on the target image to obtain a target semantic label image corresponding to the target image, where the target semantic label image includes the foreground area;

[0262] The target connected domain in the foreground area is regarded as the salient area in the target image;

[0263] The center point of the salient area is used as the initial predicted root node in the target mind map.

[0264] In one implementation, the processing unit 2102 is further configured to:

[0265] Acquire a training image, wherein the training image includes a training mind map;

[0266] Performing a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image; and

[0267] Calling the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantic label image;

[0268] Obtaining a loss function of the initial semantic segmentation model, and calculating a loss value of the loss function based on the first semantic label image and the second semantic label image;

[0269] If the loss value meets the preset conditions, the initial semantic segmentation model reaches the convergence condition and is used as the trained target semantic segmentation model.

[0270] If the loss value does not meet the preset conditions, the initial semantic segmentation model is iteratively trained.

[0271] In one implementation, the processing unit 2102 is configured to perform a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image, specifically to:

[0272] Performing region extraction processing on the training image to obtain an attention image, which includes a salient region;

[0273] Extract features from the attention image to obtain a feature map;

[0274] The category labels in the feature map are redefined according to the attention image to obtain a first semantic label image.

[0275] In one implementation, the processing unit 2102 is configured to perform region extraction processing on the training image to obtain the attention image, specifically for:

[0276] Obtaining a root node annotation file corresponding to the training image, wherein the root node annotation file includes coordinate information of the root node in the preset training mind map;

[0277] A region extraction process is performed on the training image based on the coordinate information to generate an attention image, wherein the attention image includes a salient region centered at a coordinate position indicated by the coordinate information.

[0278] In one implementation, the salient region refers to a circular region centered at the coordinate position indicated by the coordinate information. The processing unit 2102 is configured to perform region extraction processing on the training image based on the coordinate information to obtain the attention image, specifically for:

[0279] Obtain the display size information of the training image and calculate the radiation radius based on the display size information;

[0280] Determine the salient area in the training image based on the coordinate information and radiation radius;

[0281] The thermal value of each feature point in the salient region is calculated to generate an attention image; the thermal value of the feature point in the salient region in the attention image is greater than the thermal value threshold.

[0282] In one implementation, the processing unit 2102 is configured to redefine the category labels in the feature map according to the attention image to obtain the first semantic label image, specifically to:

[0283] The thermal value of each feature point in the salient area of the attention image is used to replace the label value of the category label of the corresponding feature point in the feature map to obtain a first semantic label image.

[0284] In one implementation, the processing unit 2102 is configured to call the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantic label image, specifically for:

[0285] Call the downsampling module in the initial semantic segmentation model to downsample the training image to obtain the first resolution image;

[0286] Call the N-order convolutional layer in the initial semantic segmentation model to extract features from the first resolution image to obtain N feature maps with different resolutions; N is an integer greater than 1;

[0287] Call the feature fusion module in the initial semantic segmentation model to perform feature fusion processing on N feature maps to obtain the target feature map;

[0288] The classification module in the initial semantic segmentation model is called to classify the target feature map to obtain a second semantic label image.

[0289] In one implementation, the processing unit 2102 is configured to call a feature fusion module in the initial semantic segmentation model to perform feature fusion processing on N feature maps to obtain a target feature map, specifically for:

[0290] Upsample the feature maps whose resolution is less than the resolution threshold in the N feature maps to obtain N feature maps with the same resolution;

[0291] N feature maps with the same resolution are fused to obtain the target feature map.

[0292] According to one embodiment of the present application, Figure 21The various units in the mind map recognition device shown can be individually or fully combined into one or several other units to form a whole, or one (or some) of the units can be further divided into multiple functionally smaller units to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the function of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the mind map recognition device can also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, the following can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 3 、 Figure 9 and Figure 16 A computer program (including program code) for each step of the corresponding method shown in FIG. Figure 21 The mind map recognition device shown in , and the mind map device method for implementing the embodiment of the present application. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0293] In an embodiment of the present application, the processing unit 2102 can predict the root node of the target mind map in the target image in response to the restoration operation on the target mind map, and display the predicted root node of the target mind map in the target image; this can achieve automatic prediction of the root node in the target mind map, and compared with manually marking the root node, it avoids manual intervention in the confirmation process of the root node in the mind map, thereby reducing the workload of the user object. Furthermore, the processing unit 2102 can continue to restore the complete target mind map based on the predicted root node in response to the user object's confirmation operation on the predicted root node; this restoration of the target mind map based on the accurate predicted root node obtained by prediction improves the accuracy of the restored target mind map. In the above scheme, in the restoration process of the target mind map, the complete target mind map can be quickly and accurately restored without manual intervention, thereby improving the restoration efficiency of the target mind map.

[0294] Figure 22 FIG2 shows a schematic diagram of a computer device provided by an exemplary embodiment of the present application. Figure 22, the computer device includes a processor 2201, a communication interface 2202 and a computer-readable storage medium 2203. The processor 2201, the communication interface 2202 and the computer-readable storage medium 2203 can be connected via a bus or other means. The communication interface 2202 is used to receive and send data. The computer-readable storage medium 2203 can be stored in the memory of the computer device. The computer-readable storage medium 2203 is used to store computer programs. The computer programs include program instructions. The processor 2201 is used to execute the program instructions stored in the computer-readable storage medium 2203. The processor 2201 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0295] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more instructions suitable for being loaded and executed by the processor 2201 are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0296] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 2201 loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned mind map recognition method embodiment; in a specific implementation, the one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and execute the following steps:

[0297] Get a target image containing a target mind map;

[0298] In response to a restoration operation on the target mind map, displaying a predicted root node in the target mind map in the target image;

[0299] In response to a confirmation operation on the predicted root node, a target mind map is output, and the target mind map is in an editable state.

[0300] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and, when executed to display the predicted root node in the target mind map in the target image, specifically perform the following steps:

[0301] Annotate the target image with the predicted root node in the target mind map;

[0302] The annotation display includes: displaying the predicted root node in the form of a visual display; or displaying the predicted root node in the form of an annotation mark;

[0303] The annotation mark is displayed in the area where the predicted root node is located; or, the annotation mark is displayed in the target image in the form of an annotation.

[0304] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and, when executing the process of obtaining a target image containing a target mind map, specifically perform the following steps:

[0305] Display the function selection interface, which includes the option of converting images to mind maps;

[0306] In response to triggering the image-to-mind-map option, displaying an image acquisition interface;

[0307] In the image acquisition interface, a target image containing a target mind map is acquired.

[0308] In one implementation, the image acquisition interface includes an image upload option. One or more instructions in a computer-readable storage medium are loaded by the processor 2201 and executed to acquire a target image containing a target mind map in the image acquisition interface. Specifically, the following steps are performed:

[0309] In response to triggering the image upload option, displaying at least one candidate image;

[0310] Any candidate image is selected from at least one candidate image, the selected candidate image is used as a target image, and a mind map included in the candidate image is used as a target mind map included in the target image.

[0311] In one implementation, the image acquisition interface includes an image scanning option. One or more instructions in a computer-readable storage medium are loaded by the processor 2201 and executed to acquire a target image containing a target mind map in the image acquisition interface. Specifically, the following steps are performed:

[0312] In response to triggering the image scanning option, a scanning operation is performed on the target image containing the target mind map to obtain the target image.

[0313] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and further perform the following steps:

[0314] In response to an operation of reselecting a root node in the target image, canceling the labeling display of the predicted root node in the target image, and labeling and displaying the reselected new root node in the target image;

[0315] In response to a confirmation operation on the new root node, outputting a target mind map;

[0316] The annotation display of the predicted root node is revoked to indicate that the predicted root node has not been confirmed.

[0317] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and further perform the following steps:

[0318] In response to an editing operation on a target mind map, updating the target mind map;

[0319] In response to a sharing operation on the updated target mind map, a sharing process is performed on a sharing image including the updated target mind map.

[0320] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and, when executing the restoration operation on the target mind map and displaying the predicted root node in the target mind map in the target image, specifically perform the following steps:

[0321] In response to the restoration operation on the target mind map, calling the target semantic segmentation model to perform root node prediction processing on the target image to obtain an initial predicted root node in the target mind map;

[0322] According to the initial prediction root node and each node in the target mind map, determining the node distance between the initial prediction root node and each node;

[0323] The node with the shortest distance to the initial predicted root node is used as the predicted root node.

[0324] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and executed to call the target semantic segmentation model to perform root node prediction processing on the target image to obtain the initial predicted root node in the target mind map, specifically performing the following steps:

[0325] Call the target semantic segmentation model to perform semantic segmentation processing on the target image to obtain a target semantic label image corresponding to the target image, where the target semantic label image includes the foreground area;

[0326] The target connected domain in the foreground area is regarded as the salient area in the target image;

[0327] The center point of the salient area is used as the initial predicted root node in the target mind map.

[0328] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and further perform the following steps:

[0329] Acquire a training image, wherein the training image includes a training mind map;

[0330] Performing a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image; and

[0331] Calling the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantic label image;

[0332] Obtaining a loss function of the initial semantic segmentation model, and calculating a loss value of the loss function based on the first semantic label image and the second semantic label image;

[0333] If the loss value meets the preset conditions, the initial semantic segmentation model reaches the convergence condition and is used as the trained target semantic segmentation model.

[0334] If the loss value does not meet the preset conditions, the initial semantic segmentation model is iteratively trained.

[0335] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and, when performing a first semantic segmentation process on the training mind map in the training image to obtain a first semantically labeled image, specifically perform the following steps:

[0336] Performing region extraction processing on the training image to obtain an attention image, which includes a salient region;

[0337] Extract features from the attention image to obtain a feature map;

[0338] The category labels in the feature map are redefined according to the attention image to obtain a first semantic label image.

[0339] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and, when performing region extraction processing on the training image to obtain the attention image, specifically perform the following steps:

[0340] Obtaining a root node annotation file corresponding to the training image, wherein the root node annotation file includes coordinate information of the root node in the preset training mind map;

[0341] A region extraction process is performed on the training image based on the coordinate information to generate an attention image, wherein the attention image includes a salient region centered at a coordinate position indicated by the coordinate information.

[0342] In one implementation, the salient region refers to a circular region centered at the coordinate position indicated by the coordinate information. When the processor 2201 loads one or more instructions in the computer-readable storage medium and performs region extraction processing on the training image based on the coordinate information to obtain the attention image, the processor 2201 specifically performs the following steps:

[0343] Obtain the display size information of the training image and calculate the radiation radius based on the display size information;

[0344] Determine the salient area in the training image based on the coordinate information and radiation radius;

[0345] The thermal value of each feature point in the salient region is calculated to generate an attention image; the thermal value of the feature point in the salient region in the attention image is greater than the thermal value threshold.

[0346] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and, when executing the process of redefining the category labels in the feature map according to the attention image to obtain the first semantic label image, specifically perform the following steps:

[0347] The thermal value of each feature point in the salient area of the attention image is used to replace the label value of the category label of the corresponding feature point in the feature map to obtain a first semantic label image.

[0348] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and, when executing the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantically labeled image, specifically perform the following steps:

[0349] Call the downsampling module in the initial semantic segmentation model to downsample the training image to obtain the first resolution image;

[0350] Call the N-order convolutional layer in the initial semantic segmentation model to extract features from the first resolution image to obtain N feature maps with different resolutions; N is an integer greater than 1;

[0351] Call the feature fusion module in the initial semantic segmentation model to perform feature fusion processing on N feature maps to obtain the target feature map;

[0352] The classification module in the initial semantic segmentation model is called to classify the target feature map to obtain a second semantic label image.

[0353] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 2201 and executed to call the feature fusion module in the initial semantic segmentation model, perform feature fusion processing on N feature maps, and obtain a target feature map. Specifically, the following steps are performed:

[0354] Upsample the feature maps whose resolution is less than the resolution threshold in the N feature maps to obtain N feature maps with the same resolution;

[0355] N feature maps with the same resolution are fused to obtain the target feature map.

[0356] Based on the same inventive concept, the principles and beneficial effects of solving problems by the computer device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving problems by the mind map recognition method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0357] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described mind map recognition method.

[0358] Those skilled in the art will appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0359] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data processing device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0360] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technical object that can be easily conceived by a person familiar with the technical field within the technical scope disclosed in this invention should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A mind map recognition method, characterized in that: include: Get a target image containing a target mind map; In response to the restoration operation on the target mind map, calling the target semantic segmentation model to perform semantic segmentation processing on the target image to obtain a target semantic label image corresponding to the target image, wherein the target semantic label image includes a foreground area; taking the target connected domain of the foreground area as a salient area in the target image; Taking the center point of the salient area as the initial predicted root node in the target mind map; Displaying a predicted root node in the target mind map in the target image; the predicted root node is a node in the target mind map having the shortest node distance to the initial predicted root node; In response to a confirmation operation on the predicted root node, the target mind map is output, and the target mind map is in an editable state.

2. The method according to claim 1, wherein The step of displaying the predicted root node in the target mind map in the target image includes: Marking and displaying the predicted root node in the target mind map in the target image; The annotation display includes: displaying the predicted root node in the form of a visual display; or displaying the predicted root node in the form of an annotation mark; The annotation mark is displayed in the area where the predicted root node is located; or, the annotation mark is displayed in the target image in the form of an annotation.

3. The method according to claim 1, wherein The step of obtaining a target image containing a target mind map includes: Displaying a function selection interface, wherein the function selection interface includes an option of converting an image to a mind map; In response to triggering the image-to-mind-map option, displaying an image acquisition interface; A target image including a target mind map is acquired in the image acquisition interface.

4. The method according to claim 3, wherein The image acquisition interface includes an image upload option, and acquiring a target image containing a target mind map in the image acquisition interface includes: In response to triggering the image upload option, displaying at least one candidate image; Any candidate image is selected from the at least one candidate image, the selected candidate image is used as a target image, and the mind map included in the any candidate image is used as a target mind map included in the target image.

5. The method according to claim 3, wherein The image acquisition interface includes an image scanning option, and acquiring a target image containing a target mind map in the image acquisition interface includes: In response to triggering the image scanning option, a scanning operation is performed on the target image containing the target mind map to obtain the target image.

6. The method according to claim 2, wherein The method further comprises: In response to an operation of reselecting a root node in the target image, canceling the labeling and displaying of the predicted root node in the target image, and labeling and displaying the reselected new root node in the target image; In response to a confirmation operation on the new root node, outputting a target mind map; The annotation display of the predicted root node is revoked to indicate that the predicted root node has not been confirmed.

7. The method according to claim 1, wherein After outputting the target mind map, the method further includes: In response to an editing operation on the target mind map, updating the target mind map; In response to a sharing operation on the updated target mind map, a sharing process is performed on a sharing image including the updated target mind map.

8. The method according to claim 1, wherein Before displaying the predicted root node in the target mind map in the target image, the method further includes: Determine, based on the initial predicted root node and each node in the target mind map, a node distance between the initial predicted root node and each node; The node with the shortest node distance to the initial predicted root node is used as the predicted root node.

9. The method according to claim 8, wherein The method further comprises: Acquire a training image, wherein the training image includes a training mind map; performing a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image; and Calling the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantic label image; Obtaining a loss function of the initial semantic segmentation model, and calculating a loss value of the loss function based on the first semantic label image and the second semantic label image; If the loss value meets the preset conditions, the initial semantic segmentation model reaches the convergence condition, and the initial semantic segmentation model is used as the trained target semantic segmentation model; If the loss value does not meet the preset conditions, the initial semantic segmentation model is iteratively trained.

10. The method according to claim 9, wherein The performing a first semantic segmentation process on the training mind map in the training image to obtain a first semantic label image includes: Performing region extraction processing on the training image to obtain an attention image, wherein the attention image includes a salient region; Performing feature extraction on the attention image to obtain a feature map; The category labels in the feature map are redefined according to the attention image to obtain a first semantic label image.

11. The method according to claim 10, wherein The performing region extraction processing on the training image to obtain the attention image includes: Obtaining a root node annotation file corresponding to the training image, wherein the root node annotation file includes coordinate information of a preset root node in the training mind map; A region extraction process is performed on the training image based on the coordinate information to generate an attention image, wherein the attention image includes a salient region centered at a coordinate position indicated by the coordinate information.

12. The method according to claim 11, wherein The salient region refers to a circular region centered at the coordinate position indicated by the coordinate information; performing region extraction processing on the training image based on the coordinate information to obtain the attention image includes: Obtaining display size information of the training image, and calculating the radiation radius according to the display size information; determining a salient area in the training image according to the coordinate information and the radiation radius; The thermal value of each feature point in the salient area is calculated to generate an attention image; the thermal value of the feature point in the salient area in the attention image is greater than a thermal value threshold.

13. The method according to claim 12, wherein: The redefining of the category labels in the feature map according to the attention image to obtain a first semantic label image includes: The thermal value of each feature point in the salient area in the attention image is used to replace the label value of the category label of the corresponding feature point in the feature map to obtain a first semantic label image.

14. The method according to claim 9, wherein The calling of the initial semantic segmentation model to perform a second semantic segmentation process on the training mind map in the training image to obtain a second semantic label image includes: Calling a downsampling module in the initial semantic segmentation model to downsample the training image to obtain a first resolution image; Calling an N-th order convolutional layer in the initial semantic segmentation model to perform feature extraction on the first resolution image to obtain N feature maps with different resolutions, where N is an integer greater than 1; Calling the feature fusion module in the initial semantic segmentation model to perform feature fusion processing on N feature maps to obtain a target feature map; The classification module in the initial semantic segmentation model is called to perform classification processing on the target feature map to obtain a second semantic label image.

15. The method according to claim 14, wherein The calling of the feature fusion module in the initial semantic segmentation model to perform feature fusion processing on the N feature maps to obtain a target feature map includes: Upsampling the feature maps whose resolutions are less than the resolution threshold among the N feature maps to obtain N feature maps with the same resolution; The N feature maps with the same resolution are fused to obtain a target feature map.

16. A mind map recognition device, characterized in that: include: an acquisition unit, configured to acquire a target image containing a target mind map; a processing unit configured to, in response to a restoration operation on the target mind map, call a target semantic segmentation model to perform semantic segmentation processing on the target image to obtain a target semantic label image corresponding to the target image, wherein the target semantic label image includes a foreground area; taking the target connected domain of the foreground area as a salient area in the target image; Taking the center point of the salient area as the initial predicted root node in the target mind map; Displaying a predicted root node in the target mind map in the target image; the predicted root node is a node in the target mind map having the shortest node distance to the initial predicted root node; The processing unit is further configured to output the target mind map in response to a confirmation operation on the predicted root node, wherein the target mind map is in an editable state.

17. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the mind map recognition method according to any one of claims 1 to 15 is implemented.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the mind map recognition method according to any one of claims 1 to 15.

19. A computer program product, characterized in that The computer program product includes computer instructions, and when the computer instructions are executed by a processor, the mind map recognition method according to any one of claims 1 to 15 is implemented.

Images